1 Searching an existing corpus

Transcription

1 Advanced Research Methods 2010 Mick O Donnell Week 2: Annotating, Searching and Analysing Corpora This week we will explore the use of linguistic corpora, both in terms of using a corpus you collect yourself, and also using large online corpora. 1 Searching an existing corpus We will start with how to use large existing corpora, and what to do with them. One of the classic corpora of this kind is the British National Corpora, consisting of 100 million words of various registers, collected in the period Fortunately, Mark Davies at Brigham Young University in the States has made a search interface to this corpus available over the web. Link: The webpage for this site is shown in Figure 1.. Figure 1: The BYU BNC Search Interface 1

2 1.1 A Basic Search To get started: a) Under Display, select Chart b) In the words space, put however c) Press the Search button. On the right, you should now see a bar chart, which shows how many occurrences of however were found per million words over the 5 genres included in the BNC. Note that the occurrence is far higher in Academic writing than any other genre. It is lowest in the spoken section of the corpus. Click on any of the bars in the bar chart, and to the right will be shown details about the hits in this corpus, and the corpus size. 1.2 Wildcard Search Now, change the search string to the* (note the * ) and change the Display option to List, and press Search. The table on the right now displays the most frequent 100 words which start with the, showing how often the word occurs in the corpus. Note the word the occurs much more frequently than any other word: it is the most common word in the English language. To verify this, change the search string to simply * and press search again. What is the second most common token in the corpus? What is the second most common word? What is the most common open-class word? Note that the * can appear anywhere in a search string. Try searching for con*ert * will match any number of characters (0 or more). If you want to match just 1 character, use? instead. Try searching for con?ert. See the difference in the results? 1.3 POS Search Rather than searching for words themselves, we can also search for word classes. a) Click on the POS List label in the search space. b) From the list, select verb.be. This will insert [vb*] in the search string space, which is the search pattern for all inflections of the verb to be. c) From the POS list, select verb.en. This will insert [v?n*] in the search string space, which is the pattern for all past participle verbs. d) Press Search. You will be told that there are too many hits for both search terms to show you the results. So let s be more specific. Put I in front of the search string, e.g., I [vb*] [v?n*] Now, search. Then switch to Chart mode. What can you say about the contexts in which passive clauses with a first person subject are used? Why is Fiction as strong as Spoken on this feature? Do you think this is due to the passive structure? or first person Subject? How would you verify this hypothesis? Other online search sites: Cobuild Collins: 2

3 2 Means of analyzing an unannotated corpus This section summarises the various ways you can analyse an unannotated corpus (whether your own texts or an existing one). 2.1 Simple Search counts The kinds of searches we saw above return hit counts (how many were found) and more useful, hit rate (hits per million words). By comparing the hit rates in two different subcorpora (e.g., spoken vs. written), we can the corpora differ. 2.2 Concordance A concordance is a list of all occurrences of a each word in a corpus. For example, see Figure 2. Figure 2: An example Concordance This concordance is shown in KWIC format ( Key Word In Context ). Used for: By showing you all occurrences of a word in context, one can explore the grammatical contexts for the word to appear, or the different meanings it can carry. Tools: You can perform concordance studies on your own texts. Some tools: MicroConcord: Wordsmith Tools ( may have automatic POS tagging. Not free. Concordance: UAM CorpusTool (free, automatic POS tagging for English) Concordancing with UAM CorpusTool ( allows concordance matching of your own corpus of texts. The texts do not need to be part of speech tagged: it uses a large lexicon of English to identify the part of speech of each word on the fly. Below is a diagram of a search for again any be verb followed immediately by a participle verb. 3

4 The search language is as follows (these operations are mostly available in other systems): CONCORDANCE PATTERNS: Literal token: the, however. Wildcard token: ca*, *ed, bro*en. # matches any single word. LEXICON BASED MATCHING (currently only matches any noun matches any verb matches any verb which is classified as matches noun classified as human-noun ca*@noun matches nouns starting with ca. *ing@mental-verb matches mental-verb verbs ending with ing. break% matches all inflections of root break red% matches red, reds (noun), redder (adj), reddest red%noun matches red, reds These instructions can be found under the Help window. A full list of word classes can be found by selecting Show Wordclass network from the Misc menu at the top of the screen. Examples: Find all passive clauses: be% Find modalised Find negated never 4

5 2.3 Frequency Lists Concordance tools like Wordsmith will provide you with a list of the words in your corpus, sorted for overall frequency (most frequent first), as a count, and as a percentage of all words. Used for: One can, for instance, work out what percentage of the whole corpus is accounted for by the first 1000 words. Such lists are useful for teaching, as they can highlight the words which need to be taught in a particular genre or field (e.g., what are the words most important for an engineer to know?) Tools: Wordsmith 2.4 Keywords The top words in any frequency list for English will be words such as the, of and a. A more informative listing works out how important each word is for a particular corpus, when compared with a more general corpus. For instance, the keywords from a corpus split over three fields are shown below. The words are ordered in terms of their specialness for this corpus (relative frequency in this corpus when compared to the relative frequency in the general corpus). A value of 100 indicates the word appears 100 times more in this corpus than in other corpora. Military Economics Crime economy companies stock tax cuts 85.0 profits 80.0 investment 75.0 billion 75.0 returns 70.0 sales 70.0 earnings 65.0 investors 65.0 jobs 65.0 package 65.0 assets 65.0 prices 60.0 bill 60.0 corporate 60.0 stocks markets 55.0 budget 55.0 finance 50.0 volatility 50.0 reforms 45.0 commercial 40.0 temporary 40.0 cent analysts growth troops weapons engine mountains smoke 90.0 gulf 85.0 enemy 85.0 aircraft 80.0 force 70.0 civilians 70.0 civilian 70.0 guys 65.0 military squadron 60.0 suicide 55.0 tanks 55.0 soldier 55.0 jungle 55.0 altitude 55.0 strikes 55.0 trees 55.0 lieutenant 55.0 withdrawal 55.0 missile 55.0 bomber 50.0 invasion 50.0 combat 50.0 rounds 50.0 missions 45.0 crime detective 50.0 police disappearance 40.0 criminal court justice driver boy victims 18.6 family child car lived officers legal children kids 9.3 mercy 9.3 investigators 9.3 woman 9.01 murder 8.52 boys 8.52 age 7.77 victim 6.64 street 6.27 body 6.22 incident 5.98 Used for: even more than raw frequency lists, a keyword list highlights the words which are important for teaching a particular genre/register. A word like claims might occur frequently in an economic corpus, but it is a word the language learner will meet in lots of other contexts. Tools: WordSmith, UAM CorpusTool 5

6 2.5 Collocations You might be interested in what words collocate (occur with) a particular word. For instance, what are the collocations of base? A good tool for this (30 day free trial) is called Sketch Engine ( (see Figure 2) Used For: very useful for learning the uses of a word, and also if one is building a dictionary. Figure 2: A Sketch Engine description of challenge 6

7 2.6 N-Grams (multi-word phrases, lexical bundles) Rather than looking at single-words, n-gram analysis looks for sequences of words which are common in the corpus. For instance, a list of the frequent 3-grams (sequence of 3 words) that occur in a small corpus of introductions to academic papers are shown below: in terms of 12 a set of 11 in this paper 10 the performance of 7 of the two 7 be able to 7 a number of 7 the design of 7 which can be 7 the problem of 6 ad hoc networks 6 we believe that 6 of this paper 6 terms of a 5 in section 4 5 some of the 5 in order to 5 large number of 5 that can be 5 ad hoc network 5 According to Biber (e.g., Biber and Barbieri 2007), as the corpus grows to a reasonable size (millions of words), the kinds of phrases that raise to the top don t contain lexical content as such (e.g., ad hoc networks ). Rather, they are phrases which are used to frame such meanings. We see here: in terms of, a set of, etc. Used for: while keywords tell us which words we should teach in a text, n-grams can tell us which phrasings are usefully taught. For instance, assuming we were teaching students how to write introduction sections to academic papers, we collect a corpus of such texts and produce the key n-grams for various lengths. From such a corpus, we can pick up frequent phrases such as this paper reports on or this paper/article is organized as follows. Tools: Wordsmith, UAM CorpusTool 3 Annotating your own corpus 3.1 Annotation Software Where search for words and POS is not enough, you can annotate your own text files with linguistic (or other) categories, and use your annotated corpus to perform statistical studies. You need to provide your text (or audio or video), and load them up in the annotation tool. Some annotation tools include: Text Annotation: UAM CorpusTool ( allows you to annotate multiple texts at multiple linguistic levels MMAX-2 ( annotate single text files. Worse graphical interface than CorpusTool, but allows you to annotate reference chains. Knowtator ( easy to use. Not a lot of features. Audio Annotation: Transcriber ( EXMARaLDA ( Video Annotation: Anvil ( 7

8 3.2 Annotating with UAM CorpusTool UAM CorpusTool allows you to annotate multiple texts at multiple linguistic levels, you provide the coding scheme at each level, such as that below: To annotate a text, you simply select some text, and then choose the features to assign to this segment from those presented below. See the CorpusTool manual for instructions on manual annotation of your corpus (under the Help menu at the top of the screen). 8

9 The tool also allows you to search your annotated corpus for tagged segments: 4 Statistical Analysis of an annotated corpus See the CorpusTool manual for instructions on performing statistical analyses of your annotated corpus (under the Help menu at the top of the screen). 5 References Biber, Douglas and Federica Barbieri 2007 Lexical bundles in university spoken and written registers. English for Specific Purposes 26 (2007)