Grammaticality Judgment Tests: Trial by Error

Transcription

1 Journal of Language and Linguistics Volume 5 Number ISSN Grammaticality Judgment Tests: Trial by Error Wayne Rimmer International House Moscow Abstract A standard method of determining whether a construction is well-formed is a grammaticality judgment test, where subjects make an intuitive pronouncement on the accuracy of form and structure in individual decontextualized sentences. An empirical study of the validity of grammaticality judgment tests is presented. Teachers of English (n = 52) were asked to rate the grammaticality of ten constructions which are contentious, e.g. center-embedding. The results varied with considerable agreement by the subjects on some sentences, e.g. the perceived incorrectness of Who did you quit college because you hated?, and a very divided response on others, e.g. John angered while Susan amused the woman. In evaluating the degree of (non)- consensus, there is some clash between evidence on the frequency of usage, from corpora, and internal linguistic criteria, e.g., from constraints in Universal Grammar. It is concluded that grammaticality judgment tests offer a probabilistic rather than definitive answer to the question, Is x well-formed? Introduction: the good, the bad and the marginal And don t forget, dearest lad, that the verb to be takes a nominative complement. What? You said it s them. What you meant of course was, it s they. Trefusis pulled up the handbrake and opened the door. But that s unbearable pedantry. Who, in their right mind, says It s they? No one. (Stephen Fry, The Liar, p. 311) Professor Trefusis articulates the frequent dilemma between what we should say and what we do say. Or, to be more exact, what we think we should say, as the rules for preferring one construction to another are seldom categorical. In the novel, Trefusis is an old school linguist and his frequent observations on contemporary 246

2 English are based on the grammar of Latin. There is a rule in Latin that the copula requires a complement in the nominative case, thus from John 14: 6 in the Vulgate: ego sum via, et veritas, et vita I am way-nom and truth-nom and life-nom I am the way, the truth and the life However, English is not a language with rich morphology and it has long been recognized, e.g. Jespersen (1922: 23), that grammatical analogies with Latin have little relevance. As Trefusis admits, in English the object pronoun is much more common in subject complement position, but he is wrong in implying that this is a result of ignorance or falling standards, as the phenomenon has a fully grammatical explanation. Quirk, Greenbaum, Leech and Svartvik (1985: ) explain that the default case for the pronoun outside preverbal subject position is the object form. For example, when the predicate is elided to give a short immediate response, the object pronoun is overwhelmingly preferred: Who knows the answer? - Me Our natural fascination with the right way of saying things has largely been pursued from a negative angle, spawning a preoccupation with the cause and nature of language error. There has been considerable research into error from a number of different perspectives: Davies, Criper and Howatt (1984) treat error as interlanguage; Jenkins (2005) argues for a new definition of error to reflect the increasing internationalism of English; Jones (1998) gives a sociolinguistic account of error; the collection of Richards (1984) is inspired by the influential Error Analysis methodology; Vousden and Maylor (2006) compare phonological errors in children and adults. It would be fair to say that there is more consensus on the significance of error than its identification. For instance, one critic s error is often another critic s creativity: according to Cook (2000), our natural penchant for playing with language takes us into the territory where accuracy loses its boundaries. This process is actually associated with highly-developed language competence because it demands great sensitivity to language norms and awareness of occasions when they can be eased or broken. To illustrate, consider the position of sentence a. a.?in prison she got lucky and left well alone. Coordination is usually a matter of like with like, which is not the case here as the get copula has two different complements, an adjective and participle. These constructions are incompatible as predicative complements of get according to Huddleston and Pullum (2002: 1327). Still, sentence a sounds acceptable compared to b which also mixes an adjective with a non-finite complement. b. *In prison she got lucky and to have her own cell. What makes a sound more well-formed? One possibility is that the alliteration of l in lucky left well alone makes a harmony that allows stylistic override of the rules governing what constituents can be coordinated. a could be considered as an example of zeugma, a literary term for the deliberate joining (literally yoking from Greek) of 247

3 elements that cannot be coordinated syntactically. The impression is that a, notwithstanding its syntactic marginality, seems like clever language play while b is the type of error associated with language learners. Skilled users have the ability to depart from the conventional constraints of grammar when stylistic or pragmatic needs are more pressing and this is only made possible by a thorough understanding of the rules they are negotiating (Davies 2003: 108-9). However, it would be a dangerous overreaction to deduce from such examples that all error is ultimately subjective and explicable within a social or communicative context. Jenkins for example (2006: 44) is surely wrong to defend usage such as four furnitures because it is found in emerging varieties of English. There is certainly an element of relativity in comparing errors, with some deemed more deviant than others, and the category of error is very broad and heterogeneous, as the Common European Framework (CEFR) is careful to acknowledge (Council of Europe 2001: 155). Nevertheless, the category may be fuzzy but it is not vacuous. Error is a reality of language use and dealing with grammar that does not conform to a rule-based description of the standard language presents theoretical and practical challenges. Teachers experience both because they have the difficult task of synthesizing often conflicting positions on error into a cogent and consistent model for second language learners. Two methods of identifying error, corpus analysis and intuition, are often contrasted, as discussed in the next section. Their approaches to the problem are methodologically quite different reflecting alternative theoretical bases. A tale of two methodologies A corpus is a body of texts put together in a principled way, often for the purposes of linguistic research. (Johansson 1991: 3). Computer technology means that modern corpora are large and easy to manipulate, thus lending themselves well to frequency studies of vocabulary and, to a lesser extent (Granger 2002: 21), grammar. The assumption is that there is a relationship between how frequent a construction is attested and its status in the language. Common forms and structures will be fully grammatical, rarer items dubious. For example, -ing adjectives such as alarming, gratifying, mocking form comparatives and superlatives through periphrasis not morphology (Biber, Johansson, Leech, Conrad, and Finegan, 1999: 523), hence more/most alarming instead of *alarminger/est. Frequency data confirm this rule. The only exception found in the British National Corpus is this one occurrence of boringest: it s always the boringest people, I ve never seen any of them smile, laugh, talk. There is not enough evidence to suggest that boringest is a competing form to the standard most boring. In the concordance line quoted, the speaker may want to highlight how dull these people are by contrasting his/her own interesting personality. The adoption of the marked form boringest shows individuality and an unconventional streak, distancing the speaker from the people being denigrated. But this is the domain of stylistics, not grammar. There is no question that boringest belongs to the grammar of standard English. The traditional cornerstone of corpus linguistics is the view that grammatical description is only valid if it can be demonstrated by a sufficient quantity of naturally occurring data (Sinclair 1991). The bigger the corpus, the better, with the internet representing the ultimate mega-corpus (Corpus colossal 2005). Recently there has been a shift to smaller more specialized corpora (c.f. Ghadessy, Henry and Roseberry 2001) in order to maximize the quality of the data. Consequently, there is a great deal 248

4 of emphasis on corpus design, i.e. making the sampling procedures clear and explicit (Atkins and Clear 1992). For example, the British Academic Spoken English Corpus, BASE, (Creer and Thompson 2005) is a relatively small and restricted corpus of university lectures and seminars with great care taken on the transcription and markup of the data. The relevance of frequency findings in specialized corpora like BASE, e.g. the emergence of a preterite?earnt form of earn on the learn-learnt analogy (ibid: 158), are not necessarily compromised by limitations of genre and sample size as long as the temptation to over-generalize results is resisted (O Keeffe and Farr 2003). The introspective tradition, strongly identified with generative linguistics (Sampson 1986: 234), dismisses corpus evidence as unilluminating and irrelevant. Newmayer (2003: 691) claims that native speakers are very competent at making reliable judgments about the well-formedness of language samples. This intuitive knowledge derives from the facility with which our Universal Grammar allows us to process new sentences which we have never used or encountered before (Radford 1997: 10). Consider ellipsis: although textual ellipsis is generally anaphoric (Quirk et al 1985: 900), cataphoric ellipsis of the head noun is allowed in certain environments. Anaphoric: My first attempt was terrible, my second [GAP] even worse Cataphoric: My first [GAP] was terrible, my second attempt even worse However, if the noun is post-modified, as below with the to infinitive clause to write this article, cataphoric ellipsis is not possible. worse worse My first attempt to write this article was terrible, my second [GAP] even *My first [GAP] was terrible, my second attempt to write this article even Native speakers will instinctively accept the first sentence and reject the second because they have internalized the subtle rules that license ellipsis. The only corpus we need is in our heads, the biological product of neural networks that predetermine the acquisition of grammar, unsurpassable by corpus technology. It is misleading to think that intuitive methodologies are necessarily abstract and idealistic, a common criticism of generative linguistics for example (most famously in Hymes 1972). There is a rich literature demonstrating theory being tested out by experimental studies with first and, less commonly, second language speakers. To illustrate, Marinis, Roberts, Felser and Clahsen (2005) demonstrate that nativespeakers, but not second language learners, use phrase structure based processing mechanisms to comprehend sentences with long-distance wh-extractions. In Universal Grammar (UG) a fronted wh- pronoun originates from a site adjacent to the verb it complements. The wh- constituent moves to its sentence initial position through a series of cyclical transformations that leave empty categories (e) in the surface structure: Who do you think e John says e Mary likes e? The study measured the reaction times of subjects to visually presented sentences and found that while all subjects recorded longer pauses at the ultimate lexical subcategoriser, here likes, only native-speakers posted statistically significant pauses 249

5 at the intermediate empty categories, here think and says. Marinis et al. conclude that there are qualitative differences between first and second language processing with the former making fuller use of syntactic information (c.f. Williams, Mobius and Kim 2001). This tradition of experimental study assumes a fixed version of reality with a consensus on grammaticality. It is taken for granted that error is immediately identifiable and corrigible by mentally unimpaired adult native-speakers. In fact, there can be considerable divergence of opinion on grammaticality. MacAndrew (1991: 24-29) polled native-speakers on prepositional complements. Ten sentences were presented in a multiple-choice format and subjects had to choose the correct preposition: This model makes much greater demands individual teachers a. on b. of c. with d. to There was no unanimous agreement on any of the ten sentences. In contrast, corpus linguists are more cautious about making statements regarding well-formedness. For example, Owen (1996) queries concordance lines attesting the pattern require + to be + past participle as in: Yes, your cordon pears do require to be pruned in Summer Despite the corpus evidence, Owen feels that need is more suitable than require in this construction. Attempts to expand syntactic patterns between synonymous words may not always be felicitous and, crucially, the legitimacy of a grammatical structure or form is not determined solely by its presence in a corpus for a corpus is only a sample of language in use it can never be the language (Tribble 2000: 34, emphasis in original). Language is rather an exclusive club and membership is not granted to anybody who turns up at the door. Potential members must be vetted and approved according to agreed criteria. Frequency, as discussed above, is a key criterion but also important is informed assessment of the status of the language. This is possible by asking expert users of the language, not exclusively or even necessarily native-speakers, to determine the well-formedness of select samples of language, i.e. a grammaticality judgment test. The notion expert user, from McCarthy and Carter (2001), avoids the problematic and emotive native/non-native speaker distinction (Butcher 2005; Holliday 2004) and bases authority on language competence rather than geopolitics (Park 2006). Grammaticality judgment tests complement both intuitive and corpus methodologies because they present empirical evidence for a carefully controlled number of samples. There is none of the arbitrariness of corpora, which collect much extraneous and even non-standard data, or the subjectivity of solitary introspection. This study examines the use of a grammaticality judgment test for ten contentious constructions of English, in order to evaluate the effectiveness of the methodology and the implications for those with a practical interest in the issues of grammaticality concerned, e.g. teachers. The ten sentences are presented below. Readers may wish to pause to consider their own position on the grammaticality of the sentences. 1. Who did you quit college because you hated? 2. Either you or I are wrong. 3. John angered while Susan amused the woman. 250

6 4. The plane that the pilot that the police questioned flew crashed. 5. John was bought the book. 6. The woman sitting next to the door s shoes are like mine. 7. You should lay down on the bed. 8. I bought three mouses at the computer store. 9. There s only one person who thinks of themself in that light. 10. Susan trained like she d never done before. The four research questions are as follows: (1) What degree of consensus is there from expert-users on the grammaticality of each of the ten constructions? (2) To what extent does expert-user intuition correspond to corpus data? (3) How valid is the grammaticality judgment test methodology? (4) What is the impact on pedagogical grammar? Research design The subjects were fifty-two English language teachers employed at a private language school in Moscow. All were educated to at least first degree level and/or possessed a practical teaching qualification. The nationalities of the subjects, in order of representation, were British, USA, Canadian, Russian, Australian, Irish, South African and Italian. Subjects from countries where English is not the vernacular, e.g. Russia, had been employed after checking that their oral and written language skills were equivalent to level C2 in CEFR (Council of Europe 2001). Subjects were presented the list of ten sentences before an in-house seminar. The purpose of the exercise was explained in order to reassure teachers that it was not a test of their competence with repercussions on their work. The instructions were to mark each sentence as grammatically correct or incorrect, no hedging was allowed. Test conditions were not applied, e.g. a strict time limit and spaced out seating, but teachers were told to do the exercise individually and not collaborate. Completed tasks were anonymous, again as a reassurance strategy. Processing the results was a simple matter of calculating a ratio for each sentence between correct and incorrect judgments. Results and Discussion Not one sentence illustrated complete unanimity of opinion. The response patterns varied with considerable agreement on some sentences, e.g. the incorrectness of Who did you quit college because you hated? (rejected by 94.3% of subjects), and a very divided response on others, e.g. John angered while Susan amused the woman (rejected by 56.6% of subjects). The results are presented in Table 1 with the raw number of subjects responses converted into percentages. It is important to understand that the percentages represent numbers of expert users who accept or reject the sentence as well-formed. The percentage point does not signify a degree of grammaticality. Thus, while 36.6% of the subjects consider John was bought the book to be well-formed, this is not the same as saying that John was bought the book is somehow 36.6% correct. Each sentence is also ranked in a scale, Table 2, according to 251

7 its relative perception of incorrectness. The results for each sentence are evaluated below in light of corpus data, where available, and internal linguistic evidence. Who did you quit college because you hated? (5.7% judged correct) This sentence was rejected by the vast majority of subjects and there is a theoretical rationale for its infelicity in UG. The Subjaceny Principle includes restrictions on movement which stipulate that material cannot be extracted from adverbials (see Radford 2004: 218 ff). Compare these two sentence pairs: a. We found a case of bird-flu. What did you find a case of? b. We found a case despite the quarantine. *What did you find a case despite? The wh-question counterpart to a is well-formed because it is extracted from a complement; despite the quarantine in b functions as an adverbial and extraction is not licensed. Returning to the trial sentence, because you hated is an adverbial with hated requiring an object, hated [somebody]. This object cannot be moved into a new position to form a question. The results would seem to confirm White and Genesee (1996) that native and near-native speakers show overwhelming consistency in rejecting sentences which violate the Subjacency Principle. Either you or I are wrong. (60.4% judged correct) The issue here is one of concord. For some subjects, presumably Either you or I am wrong was preferable. When the coordinated subject of a sentence contains the first or second person pronoun there is a tendency for agreement between the verb and adjacent coordinate (Quirk et al 1985: 766). Jack doesn t care if either Jill or I have fallen down/?has fallen down. Humpty will get up when either I or the king s men?comes/come. Accordingly, Either you or I am wrong is predictable. However, there are stylistic principles involved which resist the choice of am: the 1st person singular form am is felt to be awkward and the construction is likely to be avoided in monitored style (Huddleston and Pullum 2002: 509). John angered while Susan amused the woman. (43.4% judged correct) The problem is that there are two semantic roles for the subordinator while in this sentence. The first role is to introduce a time adverbial, glossed John made the woman angry at the same time as Susan amused her. Alternatively, the adverbial could be one of contrast, glossed John made the woman angry, but Susan on the other hand amused her. The latter reading is more likely because it seems incongruous for the patient simultaneously to experience two conflicting emotions. The subjects who rejected the sentence appear to have identified while as marking a time rather than contrast adverbial clause. The informal register of the sentence may have been a factor in this decision as while for contrast/concession is heavily restricted to news and academic prose (Biber et al 1999: ). To test this 252

8 hypothesis, a subordinator with a more explicit contrastive role is substituted in the same sentence to see if it becomes more felicitous. Although is appropriate as it is exclusively associated with contrast/concession (Carter and McCarthy, 2006: 45). John angered although Susan amused the woman. The sentence is intuitively more well-formed because with a different subordinator it is free of the semantic role confusion. The plane that the pilot that the police questioned flew crashed. (15.1% judged correct) The noun phrase is complex because of the center-embedding of relative clauses. [The plane [that the pilot [that the police questioned] flew]] crashed. It is widely accepted that center-embedding poses an extreme processing load (Comrie 1989: 27; Odlin 1989: 97) but this does not entail that the construction is ungrammatical. Early work in transformational grammar established a tenet that wellformedness and acceptability may diverge (Botha 1968: 21; Chomsky 1965: 11). From a corpus perspective, while center-embedding is infrequent, it is attested, even in prestigious writers like Kipling (Sampson, 1996). Clearly, center-embedding is of marginal acceptability but this does not preclude it as a possible grammatical option. John was bought the book. (39.6% judged correct) Swan (2005: 603) notes that when ditransitive verbs like buy are in the passive, the subject is usually the indirect object and his (non-corpus) illustration seems intuitively well-formed: We were all bought little presents The sentence becomes as borderline as the questionnaire example if all and little are removed.?we were bought presents To ascertain if this is lexical idiosyncrasy, compare other ditransitive verbs. I sold John the book I handed John the book I gave John the book I bought John the book John was sold the book John was handed the book John was given the book?john was bought the book In sell, hand and give, the indirect object is an ultimate beneficiary in the sense that there is a strong assumption that John will have the book at his disposal. The situation is subtly different with John was bought the book because it is more possible that John will never take possession of the book: the purchaser may change their mind about giving it to John. If dependent elements are added to the noun phrases in the buy construction, the balance shifts in favour of an ultimate beneficiary interpretation. Thus, in We were all bought little presents the fact that the speaker knows the quantity (one for all subjects) and characteristics (little) of the presents strongly suggests that 253

9 the gifts were indeed received. As it stands, John in John was bought the book is not transparent enough as an ultimate beneficiary to convince the majority of subjects that the sentence is well-formed. The woman sitting next to the door s shoes are like mine. (30.2% judged correct) There is a clear preference in corpora for longer noun phrases, as here, to incorporate a post-modifying of phrase as in The shoes of the woman sitting next to the door are like mine. In the BNC less than 2.5% of noun phrases consisting of four words or more take the genetive s suffix (Biber et al: 304). You should lay down on the bed. (54.7% judged correct) There is considerable confusion between the verbs lay and lie, complicated by the fact that lay is also the preterite form of lie. The verb phrase is intransitive here so lie, not lay which takes an object, is required. The Cambridge Advanced Learner s Dictionary takes a hard line, labeling lay for lie as a common learner error (Walter 2005: 718), while the COBUILD dictionary merely notes that this usage is informal (Sinclair 2006: 811). I bought three mouses at the computer store. (56.6% judged correct) Mouse as a technical term only came into the language recently - the first citation in the Oxford English Dictionary (Simpson 1989) is but mouse in this sense was not used widely until the 1980s so there has not been time for a clear policy on the plural to emerge. An informal survey quoted by Pinker (1999) indicates that mice is gaining the upper-hand. There s only one person who thinks of themself in that light. (47.2% judged correct) There can be little controversy over the use of they/them/their to refer to a singular person whose gender is unknown or undisclosed (see Leech and Svartvik 2002: 58). The Shorter English Dictionary (Trumble and Stevenson 2002: 3240) quotes Fitzgerald: Ask anybody for Gordon Skerrett and they ll point him out to you. A dilemma arises with the reflexive form because in themselves the plurality is double-marked, them + selves, increasing incongruity with a singular co-indexed noun, while themself appears internally inconsistent, plural them + singular self. Susan trained like she d never done before. (71.7% judged correct) Confusion between the preposition like and conjunction as is bemoaned by usage manuals going back to Fowler (2002/1926: 325). Following prescriptive grammars, as should link the main clause, Susan trained, with the comparative clause she d never done before. In fact, the functional distinction between like and as is not so clear-cut. Of course, both like and as can be prepositions, albeit with a difference in meaning. a. Joan was employed as a teacher of French 254

10 b. Joan was employed like a teacher of French In a. Joan has a job which is to teach French. In b. Joan my not know a word of French but she has a job with similar responsibilities and/or conditions. It could be argued that a similar semantic principle operates between as and like in dependent clauses. a. Joan writes as she talks b. Joan writes like she talks In a. the writing and speaking are simultaneous activities. In b. Joan has a colloquial prose style. However, the scope of this distinction needs to be justified by corpus data. It is probably premature to claim that usage of the conjunction like is more than a matter of register. A sensible position is adopted by Huddleston and Pullum (2005: 202): like + finite clause is relatively informal but synchronically and diachronically too well-established to regard as deviant. Conclusion: the jury is out Not surprisingly, in the case of contentious items, grammaticality judgment tests do not offer conclusive evidence to support the legitimacy of a specific construction. There is unlikely to be prefect agreement on which sentences are well-formed unless a very small sample size of subjects is surveyed. In evaluating the degree of (non)- consensus, there is some clash between evidence on the frequency of usage, from corpora, and internal linguistic criteria, e.g. from constraints in Universal Grammar. Subjects were not asked to justify their grammaticality pronouncements this may be a way that the methodology could be enhanced so it is impossible to know what were the factors that influenced them in the decision-making process. However, it is feasible that there were three competing motivations in rating each sentence: (1) Appeal to usage; (2) Appeal to rules; (3) Ignorance. For this first group of subjects, usage is nine-tenths of the law. If enough people, including the subject, say/write construction x, it becomes acceptable in their grammar. Thus, the extension of lay to intransitive verb phrases, You should lay down on the bed, is licensed by its frequency of use. Grammars do change over time and trends become permanent features of the standard language. For example, Hopper and Traugott (2003: 65 66) record the stages in French whereby pas evolved from a lexical item ( step ) to a fully grammaticalized negative particle ( not ). The colloquial insertion of pas into negative statements with verbs of motion, Il ne va (pas) - He doesn t go (a step), began a process resulting in the present day obligatory ne pas periphrasis for all verbs. However, in synchronical description it is difficult to say when a feature stops being register-restricted and becomes acceptable in the standard language (Ernestova 2004). Popular usage may be short-lived (Adams 2005) so an important factor is whether a construction withstands the test of time. This issue of predictive validity is not tapped by grammaticality judgment tests: as snapshots of people s reactions on a certain occasion they are not particularly sensitive to language longevity. 255

11 The second motivation, conscious appeal to rules, is psychologically very understandable. There is something comforting about rules and it is hard to relinquish them, even in the face of conflicting evidence. Rimmer (in press) quotes the following sentence from the BNC to query the rule (see Eastwood 2005: 147; Swan 2005: 610) that if is not followed by a to-infinitive. collapse or If to be in crisis means that the whole system is on the brink of total explosion, then we probably do not have a crisis. Few rules are completely comprehensive and water-tight. For this reason, Owen (1994) criticizes the COBUILD grammar (Sinclair 1990) for producing long lists of items sharing a particular grammatical feature, e.g. nouns almost always pre-modified such as eater (ibid: 21). Exceptions and problem cases abound so, according to Owen, attempts to be proscriptive are doomed. Owen does not imply that rules are useless but that they resist description. It may be significant that all the subjects in this study were language teachers and possibly more influenced by, and knowledgeable of, formal rules than lay people. Pedagogical materials (McGill 2005) and language tests (Pupura 2004: 50) are cautious about deviating from traditional description so it would be natural for teachers to be conservative. Certainly, it would be interesting to replicate the exercise with the general public, although with linguistically unaware subjects it could be expected that the third motivation, ignorance, simply not knowing whether a rule was at stake, would subsume a larger role in the results: Wray (1995) found a disturbingly low level of language awareness amongst British undergraduates sharing English as a first language. Undoubtedly, the choice of subjects is a factor which needs to be controlled. This leads to the classic reliability/validity dilemma (Hughes 1989: 42). Generally, the more homogenous the subjects, the more consistent the results of judgment tests (reliability); but incorporating a variety of variables, e.g., subjects with different language awareness, makes the results more representative (validity). To reconcile this, Granger (1998) advocates learner profiling in learner corpora, i.e. providing detailed information about the contributors including their profession, educational background and first language. Subject profiling for grammaticality judgment tests would increase their reliability and allow a posteori analysis of variables. For example, it could be determined whether age was an important trait, older subjects possibly being less tolerant of non-standard grammar. In the pronunciation surveys featuring in his dictionary, Wells (2000) includes information on the age of contributors and in some cases phonology is influenced by age-group. For instance, younger speakers (born post 1968) are more likely than older speakers to stress the second syllable in formidable (ibid: 303). An important methodological concern brought out by comparing Wells survey with this study is that in the former subjects were required to articulate the words in sentences rather than comment on their abstract pronunciation. It is advantageous to elicit a productive sample rather than a passive response because it cannot be assumed that how subjects behave in experiments is how they use grammar in real life: people who reject (computer) mouses intuitively may still ask for mouses at the hardware store. Furthermore, the use of isolated and independent sentences in grammaticality judgment tests does little to create an authentic environment for the exercise which will maximize the correspondence between test conditions and real world performance. In second language testing there is much concern with authenticity in the sense of replicating situations of genuine communication in 256

12 assessment tasks and materials (Bachman and Palmer 1996: 23-25). This has led to a dissatisfaction with traditional sentence-based test items such as multiple-choice (Spolsky 1995: 349) and the advocation of discourse-based techniques, for example cloze, which test language in context (Dastjerdi and Talebinezhad 2006; Douglas 1998). Presumably, bucking the trend in general language assessment, sentences are retained in grammaticality judgment tests for reasons of practicality and reliability: sentences are relatively easy to construct and measure; feedback is quick and quantifiable. Testing subjects reactions to the grammaticality of features of discourse would represent a new development and there are few precedents to follow (c.f. Thornbury 2005). To summarize, there is no simple answer to the question posed by Han and Ellis (1998: 10): what are grammaticality judgment tests measuring? In terms of methodology, the subject population, e.g. their language proficiency, and presentation of items, e.g., sentences vs. units of discourse, need to be considered and controlled carefully. The results obtained cannot be taken as a kind of proof but they can become a platform for further research and initiate a dialogue with other research methodologies, especially corpus linguistics which, as essentially a collection of methods (Conrad 2006: 2), invites cooperation with other approaches. Indeed, grammaticality judgment tests share one of the more disconcerting side-effects of the plethora of corpus studies, namely a lack of confidence in grammatical description. In many cases it will only be possible to maintain a probabilistic rather than definitive position on grammaticality. Absolute statements on the existence and degree of error are increasingly vulnerable, reflecting the elusive interaction between rules, usage and innovation in grammar. Finally, where does this leave second language instruction and grammar teaching? The first thing to say is that grammar is not unprincipled chaos, there are many very strong rules which can inform learners. For example, in English determiners always precede nouns, with very minor exceptions, e.g. enough as in I ve got money enough. This is a very weak exception as although Quirk et al (1985: 388) classify enough as a post-head determiner, enough is arguably a modifier in this position (Huddleston and Pullum 2002: 397). There is a lot that teachers can say intuitively about grammar with authority but perhaps the most powerful and helpful rules seem so obvious that they are left unsaid. For cases of disputed usage, where intuition fails, the corpus has been hailed as a practical resource for self-reference or classroom use with teachers and learners performing mini-investigations into language points of interest (Leech 1997; Ragan 2001; Vickers and Morgan 2005). However, corpus-based learning is no panacea and corpora will only supply some of the answers some of the time: see Chambers (2005) for a critique of what she terms corpus consultation. The status of the teacher as arbitrator and authority in questions of grammaticality is likely to remain and it is up to the individual teacher whether their regime is totalitarian, i.e. prescriptive, or pluralistic, i.e. descriptive. Regime choice is bound to ideology so teachers must take an informed stance on issues of grammaticality and be prepared to defend themselves in a changing theoretical landscape. References Adams, S. 2005, January 12th. Slangsta rap. Guardian newspaper, Society Guardian section: 8 9. Atkins, S. and J. Clear Corpus design criteria. Literary & Linguistic Computing 7/1:

13 Bachman, L. and A. Palmer Language Testing in Practice. Oxford: Oxford University Press. Biber, D., Johansson, S., Leech, G., Conrad, S. and E. Finegan Longman Grammar of Spoken and Written English. Essex: Pearson Education. Botha, R The Function of the Lexicon in Transformational Generative Grammar. Janua Linguarum, Series Maior 38. The Hague: Mouton. Butcher, C The case against the native speaker. English Today 21/2: Carter, R. and M. McCarthy Cambridge Grammar of English. Cambridge: Cambridge University Press. Chambers, A Integrating corpus consultation in language studies. Language Learning and Technology 9/2: Chomsky, N Aspects of the Theory of Syntax. Cambridge, Mass: MIT press. Comrie, B Language Universals and Linguistic Typology (2nd edition). Oxford: Basil Blackwell. Conrad, S What can corpus linguistics offer business English teachers? IATEFL BESIG Issues January 2006: 2-5. Cook, G Language Play, Language Learning. Oxford: Oxford University Press. Corpus colossal. 2005, January 22. The Economist: 75. Council of Europe Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press. Creer, S. and P. Thompson TEI mark-up of spoken language data: the BASE experience. Reading Working Papers in Linguistics 8: Dastjerdi, H. and M. Talebinezhad Chain-preserving deletion procedure in cloze: a discoursal perspective. Language Testing 23/1: Davies, A., C. Criper. and P. Howatt Interlanguage. Edinburgh: Edinburgh University Press. Douglas, D Testing methods in context-based second language research in L. Bachman and A. Cohen (eds.): Interfaces between Second Language Acquisition and Language Testing Research. Cambridge: Cambridge University Press. Eastwood, J Oxford Learner s Grammar. Oxford: Oxford University Press. Ernestova, M Effects of evolving grammar on current English language teaching in A. Pulverness (ed.): IATEFL 2004 Selections. Whitstable: IATEFL. Fowler, H A Dictionary of Modern English Usage. Oxford: Oxford University Press. (Original work published 1926). Fry, S The Liar. London: Arrow. Ghadessy, M., A. Henry and R. Roseberry Small Corpus Studies and ELT. Amsterdam: John Benjamins. Granger, S The computer learner corpus: a versatile new source of data for SLA research in S. Granger (ed.): Learner English on Computer. New York: Longman. Granger, S A bird s eye view of learner corpus research in S. Granger, J. Hung and S. Petch-Tyson (eds.): Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching. Amsterdam: John Benjamins. Han, Y. and R. Ellis Implicit knowledge, explicit knowledge and general language proficiency. Language Testing Research 2/1: Holliday, A. (2004). Plenary: non-natives and natives in A. Pulverness (ed.): IATEFL 2004 Selections. Whitstable: IATEFL. 258

14 Hopper, P. and E. Traugott Grammaticalization (2 nd edition). Cambridge: Cambridge University Press. Huddleston, R. and G. Pullum. (2002). The Cambridge Grammar of the English language. Cambridge: Cambridge University Press. Huddleston, R. and G. Pullum A Student s Introduction to English Grammar. Cambridge: Cambridge University Press. Hughes, A Testing for Language Teachers. Cambridge: Cambridge University Press. Hymes, D On communicative competence in J. Pride and J. Holmes (eds.): Sociolinguistics. Hammondsworth: Penguin. Jenkins, J. 2005, January. The ABC of ELT ELF. IATEFL Issues: 9. Jenkins, J The spread of EIL: a testing time for testers. English Language Teaching Journal 60/1: Jespersen, O Language. London: George Allen and Unwin. Johansson, S Computer corpora in English language research in S. Johansson and A. Stenstrom (eds.): English Computer Corpora. Berlin: Mouton de Gruyter. Jones, C Errors in Language Learning and Use. Essex: Longman. Leech, G Teaching and Language Corpora: a convergence in A. Wichman, S. Fligelstone, T. McEnery, and G. Knowles (eds.): Teaching and Language Corpora. New York: Longman. Leech, G. and J. Svartvik A Communicative Grammar of English. Essex: Pearson Education. MacAndrew, R English Observed. Hove: Language Teaching Publications. McCarthy, M. and R. Carter Size isn t everything: spoken English, corpus, and the classroom. TESOL Quarterly 35/2: Marinis, T., L. Roberts, C. Felser and H. Clahsen Gaps in second language sentence processing. Studies in Second Language Acquisition 27: McGill, S Teaching your grammar to suck eggs. Ih journal 18, Spring 2005: Newmayer, F Grammar is grammar and usage is usage. Language 79/4: Odlin, T Language Transfer. Cambridge: Cambridge University Press O Keeffe, A. and F. Farr Using language corpora in initial teacher education: pedagogic issues and practical applications. TOEFL Quarterly 37/3: Owen, C Corpus-based grammar and the Heineken effect. Lexicogrammatical description for language learners. Applied Linguistics 14/2: Owen, C Do concordances require to be consulted? English Language Teaching Journal 50/3: Park, Y Will nonnative-english-speaking teachers ever get a fair chance? Essential Teacher 3/1: Pinker, S Words and Rules. London: Weidenfeld and Nicolson. Purpura, J Assessing grammar. Cambridge: Cambridge University Press. Quirk, R., S. Greenbaum, S., G. Leech and J. Svartvik A Comprehensive Grammar of the English Language. Essex: Longman. Radford, A Syntactic Theory and the Structure of English. Cambridge: Cambridge University Press. Radford, A Minimalist Syntax. Cambridge: Cambridge University Press. 259

15 Ragan, P Classroom use of a systemic-functional small learner corpus in M. Ghadessy, A. Henry & R. Roseberry (eds.): Small Corpus Studies and ELT. Amsterdam: John Benjamins. Richards, J. (ed.) Error Analysis. Essex: Longman. Rimmer, W. (in press). Measuring grammatical complexity, the Gordian knot. Language Testing. Sampson, G Schools of Linguistics. London: Hutchinson. Sampson, G From central embedding to corpus linguistics in J. Thomas and M. Short (eds.): Using Corpora for Language Research. New York: Longman. Simpson, J Oxford English Dictionary. [On-line]. Available: Sinclair, J. (ed.) COBUILD English grammar. Glasgow: HarperCollins. Sinclair, J Corpus, Concordance, Collocation. Oxford: Oxford University Press. Sinclair, J. (ed.) COBUILD Advanced Learner s Dictionary (5 th edition). Glasgow: HarperCollins. Spolsky, B Measured Words. Oxford: Oxford University Press. Swan, M Practical English Usage (3 rd edition). Oxford: Oxford University Press. Thornbury, S Beyond the Sentence: Introducing Discourse Analysis. Oxford: Macmillan. Tribble, C Practical uses for language corpora in ELT in P. Brett & G. Mottram (eds.): A Special Interest in Computers. Whitstable, IATEFL.. Trumble, W. and A. Stevenson. (eds.) (2002). Shorter English Dictionary (3 rd ed., Vol. 2). Oxford: Oxford University Press. Vickers, C. and S. Morgan Incorporating corpora. English Teaching Professional 41: Vousden, J. and E. Maylor Speech errors across the lifespan. Language and Cognitive Processes 21/1-3: Walter, E. (ed.) Cambridge Advanced Learner s Dictionary (2nd edition). Cambridge: Cambridge University Press. Wells. J Longman Pronunciation Dictionary. Harlow: Pearson Education. White, L. and F. Genesee How native is non-native? The issue of ultimate attainment in adult second language acquisition. Second Language Research 12/3: Williams, J., P. Mobius and C. Kim Native and non-native processing of English wh-questions: parsing strategies and processing constraints. Applied Psycholinguistics 22/4: Wray, A The occurrence of occurance and alot of other things aswell : patterns of errors in undergraduate English in G. Blue and P. Mitchell (eds.): Language and Education Clevedon: Multilingual matters. 260

16 Table 1: Grammaticality judgment test results for ten sentences correct (%) incorrect (%) Who did you quit college because you hated? Either you or I are wrong John angered while Susan amused the woman The plane that the pilot that the police questioned flew crashed. John was bought the book The woman sitting next to the door s shoes are like mine. You should lay down on the bed I bought three mouses at the computer store There s only one person who thinks of themself in that light. Susan trained like she d never done before Table 2: Ranking by increasing perception of grammatical error Considered most incorrect flew crashed. mine. Who did you quit college because you hated? The plane that the pilot that the police questioned The woman sitting next to the door s shoes are like John was bought the book. John angered while Susan amused the woman. that light. There s only one person who thinks of themself in You should lay down on the bed. I bought three mouses at the computer store. Either you or I are wrong. Considered most correct Susan trained like she d never done before. 261