Extending and Complementing the Common European Framework

Transcription

1 Extending and Complementing the Common European Framework John H.A.L. DE JONG "Let's not be stupid" Richard Deacon 1

2 2

3 3

4 4

5 5

6 6

7 7

8 8

9 9

10 Survey results 'Language testing in Europe: Time for a new framework?' EU North America Asia South America Middle East Australia Africa

11 Canada UK France Spain Italy Turkey Netherlands Sweden Finland Cyprus Greece Czech Republic Belgium Ireland Germany Switzerland Romania Portugal Poland Luxembourg Bulgaria

12 Standards & Measurement 12

13 13

14 14

15 100 Cool Cool 0 15

16 16

17 17

18 The Framework 18

19 Quality Language Development Two basic dimensions Quantity How many different tasks? How effective? How efficient? 19

20 Combining Quantity & Quality: 20

21 Slippery Slippery slope slope 21

22 C2 C1 B2 B1 A2 A1 Below A1 22

23 A quote from the CEF: ( ) all knowledge of a language is partial, however much of a mother tongue or native language it seems to be. ( ) In addition, a given individual never has equal mastery of the different component parts of the language in question, for example, of oral and written skills, or of comprehension and interpretation compared to production skills. (CEF, page 169; emphasis added, JdJ). 23

24 Profiled Development 24

25 The Scale 25

26 Increasing language ability & CEF Descriptors C1 C2 B2 A1 1 A2 2 3 B1 <A1 Start learning Learner Descriptor CEF boundaries Perfect 26

27 CEF Descriptors A1 A2 B1 B2 C1C2 A1 A2 B1 B2 C1 C2 A1 A2 B1 B2 C1 C2 A1 A2 B1 B2 C1 C2 Listen Speak Read Write 27

28 Equating PISA Reading and PTE Academic Reading 28

29 CEF C2 C1 B2 B1 A2 Some disadvantages: People don t all agree on the exact definition of what it means to be at a level Levels are too wide to be of practical use in measuring progress Biased set of descriptors A1 29

30 CEF GSE C2 C1 90 The scale of the GSE makes up for the disadvantages of the CEF: B2 B2 GSE 59 An exact psychometric definition of what it means to be at a level B1 Granularity to allows for fine and precise measurement A2 A1 Each score has a known correspondence to the CEF scale of levels. 10 First used with the Pearson Test of English Academic (PTE Academic). 30

31 From Task Difficulty to Learner Ability 31

32 Table 1: CEF Cut-offs (North, 2000) CEF Level Difficulty cut-off C C B B A A

33 How to define a level: a level is not a point The levels of the CEF were defined operationally as a summary of the descriptors for each of the levels This means that a level is defined by all of the descriptors at a level. For a person to be considered to be at a level therefore, they must have a probability of being able to perform 50% of the entire set of descriptors defining that level. 33

34 Table 1: CEF Cut-offs (North, 2000) CEF Level Difficulty cut-off C C B B A A Easiest A1 Descriptor: Most difficult A1 Descriptor: Average A1 descriptor:

35 Average A1 descriptor: Rasch formula To have a 50% probability of success on the set of descriptors defining A1, the required ability is At an ability of (the cut-off for A2) the probability for success at the set of descriptors of A1 will have grown to 82% 35

36 Tasks at level Table 2: Cut-offs for task difficulty and learner ability at CEF levels CEF Level C2 C1 B2 B1 A2 A1 Minimum Task Difficulty Minimum Learner Ability Table 3: Probability of success at CEF levels for learners at different levels Learners at level A1 A2 B1 B2 C1 C B B A A

37 Adding Descriptors 37

38 Adding Descriptors: Calibrating learning objectives Expert judges assign GSE values to learning objectives Sources Syllabuses New descriptors Functions Grammar Expert panels Course materials Frameworks 43 CEFR Pearson syllabus database 38

39 Rating new descriptors General Adult Syllabus 89 experienced course-ware developers from Pearson based in 10 countries across the world 316 teachers from + 50 countries with a detailed knowledge of the CEFR and a minimum of two years teaching experience Pearson staff assigned PSE scale values (10 90) teachers classified the descriptors at one of the CEF levels (A1 C2) The average classifications from the teachers were then projected onto the PSE John de Jong, Liverpool, April 2013

40 HOW TO SCALE RATINGS Ratings from judges: Level x1 Level x+1 2 Level 3 x+2 1: Close to 100% say Level 1 : object is well in the middle of Level 1 Mathematically: (1.00*1) = = 1.5 2: 70% say Level 1 and 30% say Level 2 object is close to top of Level 1 Mathematically: (0.7*1+ 0.3*2) = = 1.8 3: 30% say Level 1 and 70% say Level 2 object is close to bottom of Level 2 Mathematically: (0.3*1+ 0.7*2) = = 2.2 John de Jong, Liverpool, April 2013

41 Teacher ratings Two independent ratings of new descriptors 90 C2 80 C1 70 B B1 40 A2 30 A1 20 <A1 10 r = C2 C1 B2 B1 A2 A1 <A Pearson staff ratings John de Jong, Liverpool, April 2013

42 Pearson ratings 2012 Compare anchor items Anchor items Original calibrations (North 2000) North B2 Pearson C1 Anchor Can follow items the show a essentials high level of lectures, Can use a limited of agreement talks and reports with and number of cohesive other forms of the devices academic/professional original to link his/her calibrations utterances into from clear, coherent discourse, North propositionally (2000) and North B2 Pearson B1 presentation which are though there may be linguistically complex. some "jumpiness" in a long contribution. After removing two outliers the correlation is 0.97 John de Jong, Liverpool, April 2013

43 Global Scale of English GSE Brian North Brian North John de Jong, Liverpool, April 2013

44 This was about, as an example, The General Adult Syllabus Other Syllabuses under development: Academic Syllabus Professional Syllabus Young Learners Syllabus John de Jong, Liverpool, April 2013

45 Extending and Complementing the Common European Framework The Global Scale of English extends and complements the CEFR by providing Coverage of skills and levels where CEFR descriptors are sparse Detailed description in relation to learning objectives for English Granular evidence of proficiency development within CEF levels Precise placement of descriptors within CEFR levels John de Jong, Liverpool, April 2013