THE CEFR & UNIVERSITY ENTRANCE TESTS. a state of affairs in Europe

THE CEFR & UNIVERSITY ENTRANCE TESTS a state of affairs in Europe Bart Deygers & Cecilie Hamnes Carlsen EALTA Copenhagen, 28 May 2015

RESEARCH QUESTIONS RQ1 RQ2 RQ3 What are the language demands in European universities towards L2 students? What is the impact of the CEFR on university entrance policies and tests across Europe? What are the test developers opinions about (a) the CEFR in the context of language testing and (b) about the language testing policy in their country?

METHODOLOGY Guiding principle: purposeful selection (Freeman, 2000) Goal random sample Goal = knowledgeable & information-rich respondents (Reybold et al., 2013) Respondent selection Main criteria: ALTE member and/or EU member state Structured interviews 3 parts (national policy, test-specific, personal views) Conducted via phone / skype, recorded using Audacity Coding & analysis Double coding using NVivo10 & MS Excel

INTERVIEW QUESTIONS Is knowledge of the official language(s) of your country a study requirement? Is there one, or are there different, language requirements for university entry? Who decides what the university language requirement should be? In which ways can entering students prove that they possess the required proficiency? Are the university entrance tests in your country centrally developed or are locally developed tests also accepted? What are the entry level requirements based on? Are you aware of an empirical ground for them? Does your test grant access to university education? In what way has the CEFR influenced your test? How do you feel about the CEFR? If you could change anything in the (use of the) CEFR what would it be? Do you think the B2 level is or would be enough to function academically? Do you think B2 is operationalized in the same way throughout Europe? If you could change anything about the university entrance policy in your country, what would it be?

METHODOLOGY 31 respondents / 27 countries Austria, Belgium (Flanders), Belgium (Wallonia) Bulgaria, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Romania, Slovenia, Spain, Sweden, Switzerland, Turkey, United Kingdom

FINDINGS

WE LIKE THE CEFR Reported effects from using the CEFR: > From disharmony to standardisation R I R in 98, when we started to build the test, we started with the [first draft of the] CEFR descriptors. And it was because this test we had was so bad. I really don t know what it was testing. Okay. So you started with the CEFR before it was actually the blue book that we know now? Yes, absolutely. So we decided to create something more accurate and more efficient (R9)

WE LIKE THE CEFR Reported effects from using the CEFR: > From disharmony to standardisation > More focus on skill testing use of the language, grammar, syntax [ ] disappeared with the CEFR. The levels were different. Everything was changed really. All levels had to be changed according to the CEFR (R25)

WE LIKE THE CEFR Reported effects from using the CEFR: > From disharmony to standardisation > More focus on skill testing > Adopting a new level structure It was four levels [ ] They were called A B C D. Alpha, beta, gamma, delta. That was it. So these four levels became six now, according to the CEFR frameworks (R12)

WE LIKE THE CEFR Reported effects from using the CEFR: > From disharmony to standardisation > More focus on skill testing > Adopting a new level structure But we regret uninformed use B2 B2 B1 A2

HOW L2 STUDENTS GET IN In most contexts, there is more than one test available. Often, local tests exist alongside centralised ones Multiple central and local tests 11 Multiple central tests 5 One central test 3 Multiple local tests 3 No test 2 University entrance exam 2 Secondary school leaving exam 1

HOW L2 STUDENTS GET IN In most contexts, there is more than one test available. Often, local tests exist alongside centralised ones Often lack of standardization or control (min. 10 countries) R5 One claims that it s at B2 level, but I, sort of, can t see it. I don t know how, in which aspects. I Is it lower or higher, you think? R5 Erm. I wouldn t say higher or lower. It s just a bit of a mess they don t apply piloting, pretesting and statistical analysis, so it s not astonishing that the results were very heterogeneous and the exams of different level of difficulty (R10)

WHO DECIDES? 2 No requirement 4 Government (General rule) & university (specifics) 7 Government, ministry etc. 14 University, Faculty etc.

WHICH LEVEL? In most contexts B2 is the required A2 & B2 1 entry level B1 1 B2 10 B2+ 1 B2 & C1 8 C1 2 Not related to CEFR 2 No requirement 2

WHY THIS LEVEL? In most cases the required CEFR threshold for university entrance was based on examples of other countries rather than research Needs analysis Link with study success 3 1 Empirical research Following other institutions example (6) Literature study (6) Unknown (6) CEFR (3) 21

WHY THIS LEVEL? Little empirical foundation for the level required. R20 I R20 it was a political decision that it had to be this level for everybody. Would I be correct in saying that a political decision is not necessarily an empirical decision? Yeah, that would be correct they asked us about our opinion, and we said, in Europe it s mostly B2. B2 covers this and this. And they said, okay, it should be B2 (R24)

BUT? (3) Is B2 enough to function academically? needs Depends on Not just like that No (10) B2+ (4)

BUT? Do you think we operationalize B2 in the same way throughout Europe? No (14)

LEVELS AS NORMS I R12 I R12 Has the CEFR has influenced [your test]? Not influenced, it has copied all the parts [ ] It s the foundation, it s founded on that. We had tests before, I have all the presidential decrees here and now the announcement is that it is CEFR compatible in all levels [ ] It was four levels But they weren t C1 called B2 B1 probably? B2 No, they were called A B C D. Alpha, beta, gamma, delta. That was it. So these four levels became six now, according to the CEFR frameworks. B1 A2

LEVELS: USE/MISUSE Level requirements may shift if we need certain students. -- The situation for international students is different depending on which country they come from. If they are from the European union they are not obliged to have B2 (R4)

LEVELS: USE/MISUSE Level requirements may shift if we need certain students. -- they informed us from the ministry, that they needed the people within the health system and the C1 level was too difficult to pass (R6)

LEVELS: USE/MISUSE Level requirements may shift if we need certain students. -- we have an institute working with oil and there we don t ask any level, because we need people working with oil (R9)

LEVELS: USE/MISUSE Level requirements may shift if we need certain students. -- they will follow their own internal policy, whether they want to have a student of this kind, or whether they don t (R21)

CONCLUSIONS RQ1 What are the language demands in European universities towards L2 students? Mostly B2 Both centrally developed & local tests Admission decisions are mostly taken at university/faculty level

CONCLUSIONS RQ1 What are the language demands in European universities towards L2 students? Mostly B2 >>> but with little empirical foundation Both centrally developed & local tests >>> often leads to disharmonious policies Admission decisions are mostly taken at university/faculty level >>> so the decision makers are also the stakeholders

IMPLICATIONS In professional organisations CEFR levels are uniformly interpreted >>> This study (and previous CEFR SIG studies) do not point in that direction Non-generic specifications would lead to a normative use >>> But the CEFR is used normatively The level descriptors were meant as illustrations, not as standards >>> But in language testing they have become standards The CEFR stimulates discussion between decision makers & language experts >>> The CEFR also seems to stimulate simple level-based thinking

ACKNOWLEGEMENTS All respondents for their time, effort and input ALTE for the support Beate Zeidler & Dina Vilcu for the wonderful teamwork

THANK YOU