Does the CEFR reflect SLA-development? a question of construct validity Cecilie Carlsen Norsk språktest/lle (University of Bergen) EALTA conference Innsbruck 31.05-03.06.2012
Norwegian profile Work in progress April 2011- September 2012 8 project members 5 language test developers (Norsk språktest) 3 SLA-researchers (University of Bergen) Reference Level Descriptions (RLD) for Norwegian Financial support: Ministry of Children, Equality and Social inclusion Anthology by Desember 2012
Background Practical need: CEFR is widely used in Norway CFER is general, not language-specific Need for RLDs for Norwegian Research interest: Do the CEFR-descriptors mirror second language development? Are the CEFR-descriptors supported by empirical SLA-data? Means: ASK- electronic learner corpus linked to the CEFR
Setting the scene
[a] language test is only as good as the theory of language on which it is based McNamara 2000
What we need to know if we want to develop good [proficiency] scales is [ ] how somebody acquires language, that is, what the developmental stages in language acquisition are. (de Jong 1988:74)
The relation between language learning and proficiency levels Proficiency levels are constructions (langauge assessment) Language learning is a gradual process (SLA-research)
CEFR B2+ Can express him/herself clearly and without much sign of having to restrict what he/she wants to say. B1 Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation and circumlocutions [ ] A2 Has a repertoire of basic language which enables him/her to deal with everyday situations with predictable content [ ]
The CEFR is not a theory of SLA, but... The wording of [the CEFR] standards is important, as they represent the test CONSTRUCT, the assumed view of language proficiency which is assessed in the test and which is the target of measurement in any individual case McNamara 2011: 2
It is therefore problematic that [ ] the CEFR levels are neither based on empirical evidence taken from L2-learner performance, nor on any theory in the fields of linguistics or verbal communication (Alderson, 2007, Hulstijn 2007) Hulstijn, Alderson, Schoonen 2010:15
SLATE (Second Language Acquisition and Testing in Europe) Overall research question (2006) Which linguistic features of learner performance (for a given target language) are typical at each of the six CEFR levels? Hulstijn, Alderson, Schoonen 2010:17
How can this question be investigated? Descriptive approach - «What characterizes B1-level grammar, vocabulary, spelling etc?» Hypothesis-driven approach - «Are the predicitons of the CEFR-descriptors supported by empirical data?»
http://gandalf.aksis.uib.no/ask/ Computer learner corpus of Norwegian Collaborative effort (Norsk språktest, SLA-researchers at UIB, og AKSIS (now: UniDigital) Project leader: Kari Tenfjord, LLE, UiB Texts from two standardized tests of Norwegian for adult immigrants 1700 texts (ca. 70 000 words) Learners with 10 different L1s Automatically tagged for grammatical traits Manually tagged for errors 1222 of the texts were reassessed using the CEFR-scale A1-C2 (5-10 raters, Carlsen & Kaftandjieva 2009, part of the project ASKeladden at UiB)
Appetizers - some results from Norwegian profile
Participants and projects Grammar: Ann Kristin Helland: Verbs and past tense Marte Nordanger: The nominal clause and definiteness Cecilie Carlsen/Lisbeth Salomonsen von Mehren: Missing subjects Snorre Karkkonen Svensson: Modality Eli Moe: Syntax
Vocabulary Rønnaug Totland: Vocabulary Discourse Cecilie Carlsen: Coherence and connectives Karoline Haugsvær: Language functions Errors Trinelise Eriksson: Errors
Marte Nordanger: Grammatical marking of definite reference
Background In Norwegian nouns are grammatically marked for indefinite and definite reference. Definite sg. and definite pl. are marked with suffix -en, -ene. Indefinite sg. is marked with an article: en (M), ei (F), et (N) Indefinite sg. Definite sg. Indefinite pl. Definite pl. En gutt Gutten Gutter Guttene
The study focuses on bare noun phrases with definite reference. (Kongen heter Harald. vs. Den greie kongen vår heter Harald. The king is named Harald. vs The kind king of ours is named Harald. ) The hypotheses are based upon the CEFR-scale for Grammatical correctness. Grammatical marking of definiteness is a core category of Norwegian noun morphology.
Hypotheses H1: Correct grammatical marking of definite reference will increase from level B1 to level C1. H2: The marking of definite reference at level C1 will be more target language like than at levels B1 and B2. The marking of definite reference is a distinguishing criterion between levels B1, B2 and C1. The study also aims at investigating L1 as a distinguishing factor between English and Russian L1-groups.
The study shows: Average percentage incorrect use
Conclusions and reflections The correctness of grammatical marking of definite reference does increase from level B1 to C1 (as implicitly predicted by the CEFR). L1 is an important variable at levels B1 and B2. This makes it difficult to predict a general level of correctness for differnet L1-groups for grammatical marking of definite reference for levels B1 and B2. The L1-variable seems to be less important at level C1 (as predicted by the CEFR). Precautions: The individual variation is considerable within all levels. The number of texts at level C1 is too low to draw definit conclusions.
Rønnaug Katharina Totland: Vocabulary
Research question: Is there a measurable difference in the vocabulary from level A2 to B2 concerning SIZE (number of lemmas) and CONTROL (number of mistakes)? No expectations of surprising findings. Obvious that the vocabulary grows from a low to a high level; Therefore a descriptive method rather than a hypothesis driven method
Predictions in the CEFR Vocabulary range from A1 to C2 A1:Has a basic vocabulary repertoire of isolated words and phrases related to particular concrete situations. C2: Has a good command of a very broad lexical repertoire including idiomatic expressions and colloquialisms; shows awareness of connotative levels of meaning. Vocabulary control from A2 to C2 A2: Can control a narrow repertoire dealing with concrete everyday needs. C2: Consistently correct and appropriate use of vocabulary. CEFR p.112
Precautions: ASK is not well-suited for investigating changes in vocabulary because: There is an uneven number of texts at different proficiency levels The texts are of unequal length The texts deal with different topics (influences the vocabulary chosen. According to prior research: People tend to use the most frequent words, even when writing scientific papers/texts. In most texts about 50 % of the total vocabulary consist of the 50 most frequent words
Size of vocabulary A2 B1 B2 C1 Number of lemmas
Comparison of the most frequent words at each level + Norwegian frequency list. Small differences Curiosity: Jeg («I») is the most frequently used word at A2 B1 in 4th place B2 in 16th place C1 in 22nd place
How many lexical mistakes? Number of mistakes in the category «Missing word» Number of mistakes in the category «Wrong word» 450 400 350 300 250 200 150 100 50 0 400 350 300 250 200 150 100 50 0
Trinelise Eriksson: Errors
Background CEFR contains mainly positive formulation of what learners «can do» Often necessary to know what learners cant do as well; «can t dos», (Granger 2005) Learners errors yield important information about the learning process
Error distribution in Norwegian L2 texts according to the CEFR Hypothesis 1: The total amount of errors will be lower at the higher CEFR-levels Hypothesis 2: The types of errors will differ between the CEFR-levels
Total amount of errors
Error codes in ASK
Cecilie Carlsen/Lisbeth Salomonsen von Mehren: Missing subjects
Background Explicit subjects are compulsory in Norwegian. Even when semantically redundant The formal subject, «det» (it) is used where there is no real subject +/- compulsory subject is a distinguishing factor for the languages of the world - compulsory subject is typologically common all the roman languages (except modern French), Polish, Arabic, Vietnamese etc. Compulsory subject is relatively hard to acquire for learners with an L1 without compulsory subjects
H1: Learners at lower levels of proficiency have more sentences without explicit subject than learners at higher levels of proficiency. H1 supported by the data. The most significant differenc is between the levels A2 and B1. Only minor differences between the levels above B1.
H2: Learners with an L1 without compulsory subjects have more sentences without explicit subject in their interlanguage than learners with an L1 in which subjects are compulsory. H2 supported by the data. Norwegian interlanguage texts written by German and English learners, have only a few sentences without subjects, while texts written by learners of an L1 without compulsary subjects have many more cases of missing subjects in their interlanguage.
Preliminary conclusions The CEFR scales are to a large degree supported by empirical data in our study Some scales need to be adjusted (for Norwegian at least) Developing RLD is a time-consuming enterprise One important finding: the CEFR scales vary not ONLY according to the TL but also according to learners L1 Important to look into the L1-TL combination in relation to the CEFR as well (a point also made by Hulstijn, Alderson and Schoonen 2010 SLATE publication)
Thank you!