Extending and Complementing the Common European Framework John H.A.L. DE JONG "Let's not be stupid" Richard Deacon 1
2
3
4
5
6
7
8
9
Survey results 'Language testing in Europe: Time for a new framework?' EU North America Asia South America Middle East Australia Africa 0 10 20 30 40 50 60 70 10
Canada UK France Spain Italy Turkey Netherlands Sweden Finland Cyprus Greece Czech Republic Belgium Ireland Germany Switzerland Romania Portugal Poland Luxembourg Bulgaria 0 2 4 6 8 10 12 14 16 11
Standards & Measurement 12
13
14
100 Cool Cool 0 15
16
17
The Framework 18
Quality Language Development Two basic dimensions Quantity How many different tasks? How effective? How efficient? 19
Combining Quantity & Quality: 20
Slippery Slippery slope slope 21
C2 C1 B2 B1 A2 A1 Below A1 22
A quote from the CEF: ( ) all knowledge of a language is partial, however much of a mother tongue or native language it seems to be. ( ) In addition, a given individual never has equal mastery of the different component parts of the language in question, for example, of oral and written skills, or of comprehension and interpretation compared to production skills. (CEF, page 169; emphasis added, JdJ). 23
Profiled Development 24
The Scale 25
Increasing language ability & CEF Descriptors C1 C2 B2 A1 1 A2 2 3 B1 <A1 Start learning Learner Descriptor CEF boundaries Perfect 26
CEF Descriptors 90 80 70 60 50 40 30 20 10 0 A1 A2 B1 B2 C1C2 A1 A2 B1 B2 C1 C2 A1 A2 B1 B2 C1 C2 A1 A2 B1 B2 C1 C2 Listen Speak Read Write 27
Equating PISA Reading and PTE Academic Reading 28
CEF C2 C1 B2 B1 A2 Some disadvantages: People don t all agree on the exact definition of what it means to be at a level Levels are too wide to be of practical use in measuring progress Biased set of descriptors A1 29
CEF GSE C2 C1 90 The 10-90 scale of the GSE makes up for the disadvantages of the CEF: B2 B2 GSE 59 An exact psychometric definition of what it means to be at a level B1 Granularity to allows for fine and precise measurement A2 A1 Each score has a known correspondence to the CEF scale of levels. 10 First used with the Pearson Test of English Academic (PTE Academic). 30
From Task Difficulty to Learner Ability 31
Table 1: CEF Cut-offs (North, 2000) CEF Level Difficulty cut-off C2 3.90 C1 2.80 B2 0.72 B1-1.23 A2-3.23 A1-4.29 32
How to define a level: a level is not a point The levels of the CEF were defined operationally as a summary of the descriptors for each of the levels This means that a level is defined by all of the descriptors at a level. For a person to be considered to be at a level therefore, they must have a probability of being able to perform 50% of the entire set of descriptors defining that level. 33
Table 1: CEF Cut-offs (North, 2000) CEF Level Difficulty cut-off C2 3.90 C1 2.80 B2 0.72 B1-1.23 A2-3.23 A1-4.29 Easiest A1 Descriptor: -4.29 Most difficult A1 Descriptor: -3.22 Average A1 descriptor: -3.77 34
Average A1 descriptor: -3.77 Rasch formula To have a 50% probability of success on the set of descriptors defining A1, the required ability is -3.77 At an ability of -3.23 (the cut-off for A2) the probability for success at the set of descriptors of A1 will have grown to 82% 35
Tasks at level Table 2: Cut-offs for task difficulty and learner ability at CEF levels CEF Level C2 C1 B2 B1 A2 A1 Minimum Task Difficulty 3.90 2.80 0.72-1.23-3.23-4.29 Minimum Learner Ability - 3.35 1.76-0.26-2.24-3.77 Table 3: Probability of success at CEF levels for learners at different levels Learners at level A1 A2 B1 B2 C1 C1 0.00 0.00 0.03 0.17 0.50 B2 0.00 0.02 0.12 0.50 0.83 B1 0.03 0.12 0.50 0.88 0.97 A2 0.18 0.50 0.88 0.98 1.00 A1 0.50 0.82 0.97 1.00 1.00 36
Adding Descriptors 37
Adding Descriptors: Calibrating learning objectives Expert judges assign GSE values to learning objectives Sources Syllabuses New descriptors Functions Grammar Expert panels Course materials 45 44 43 42 Frameworks 43 CEFR Pearson syllabus database 38
Rating new descriptors General Adult Syllabus 89 experienced course-ware developers from Pearson based in 10 countries across the world 316 teachers from + 50 countries with a detailed knowledge of the CEFR and a minimum of two years teaching experience Pearson staff assigned PSE scale values (10 90) teachers classified the descriptors at one of the CEF levels (A1 C2) The average classifications from the teachers were then projected onto the PSE John de Jong, Liverpool, April 2013
HOW TO SCALE RATINGS 1 2 3 4 5 6 1 2 3 Ratings from judges: Level x1 Level x+1 2 Level 3 x+2 1: Close to 100% say Level 1 : object is well in the middle of Level 1 Mathematically: (1.00*1) + 0.5 = 1.0 + 0.5 = 1.5 2: 70% say Level 1 and 30% say Level 2 object is close to top of Level 1 Mathematically: (0.7*1+ 0.3*2) + 0.5 = 0.7+0.6 + 0.5 = 1.8 3: 30% say Level 1 and 70% say Level 2 object is close to bottom of Level 2 Mathematically: (0.3*1+ 0.7*2) + 0.5 = 0.3+ 1.4 + 0.5 = 2.2 John de Jong, Liverpool, April 2013
Teacher ratings Two independent ratings of new descriptors 90 C2 80 C1 70 B2 60 50 B1 40 A2 30 A1 20 <A1 10 r = 0.981 C2 C1 B2 B1 A2 A1 <A1 10 20 30 40 50 60 70 80 90 Pearson staff ratings John de Jong, Liverpool, April 2013
Pearson ratings 2012 Compare anchor items 90 80 70 60 50 40 30 20 10 Anchor items -5-4 -3-2 -1 0 1 2 3 4 5 Original calibrations (North 2000) North B2 Pearson C1 Anchor Can follow items the show a essentials high level of lectures, Can use a limited of agreement talks and reports with and number of cohesive other forms of the devices academic/professional original to link his/her calibrations utterances into from clear, coherent discourse, North propositionally (2000) and North B2 Pearson B1 presentation which are though there may be linguistically complex. some "jumpiness" in a long contribution. After removing two outliers the correlation is 0.97 John de Jong, Liverpool, April 2013
Global Scale of English GSE -4-3 -2-1 0 1 2 4-4 -3-2 -1 0 1 2 3 4 Brian North 2000-4.29-3.23-1.23 0.72 2.80 3.90 Brian North 2000-4.29-3.23-1.23 0.72 2.80 3.90 John de Jong, Liverpool, April 2013
This was about, as an example, The General Adult Syllabus Other Syllabuses under development: Academic Syllabus Professional Syllabus Young Learners Syllabus John de Jong, Liverpool, April 2013
Extending and Complementing the Common European Framework The Global Scale of English extends and complements the CEFR by providing Coverage of skills and levels where CEFR descriptors are sparse Detailed description in relation to learning objectives for English Granular evidence of proficiency development within CEF levels Precise placement of descriptors within CEFR levels John de Jong, Liverpool, April 2013