Measuring what really matters ways of impact measurement of employability and the effects of mobility. Uwe Brandenburg CHE Consult GmbH January 2016

Transcription

1 Measuring what really matters ways of impact measurement of employability and the effects of mobility Uwe Brandenburg CHE Consult GmbH January 2016

2 Connection to IRIS project 2

3 IRIS project Improving internationalisation of Israeli colleges Role of CHE Consult Cross-checking status quo assessment Developing indicators for internationalisation Creating a measuring system Producing individual roadmaps for the colleges to implement internationalisation procedures and structures Measuring Internationalisation Uwe Brandenburg January

4 1. Background 4

5 Internationalisation definition of term itself and application changed considerably Today, standard definition used: Internationalisation at the national/sector/ institutional levels is defined as the process of integrating an international, intercultural or global dimension into the purpose, functions or delivery of higher education. (Knight 2009, 7) Measuring Internationalisation Uwe Brandenburg January

6 Internationalisation buzzword of at least the last decade and a half started off in political science and governmental relations, entered higher education in 1980s last two decades: concept of the internationalisation moved from fringe to core of institutional agenda Effects: tends to become a conditio sine qua non creates an atmosphere of high risk of lip service and give to the emperor Measuring Internationalisation Uwe Brandenburg January

7 Internationalisation Key terms: Process incorporating global aspects into HE Differentiation between internationality and internationalisation (Brandenburg & Federkeil 2007) Value-laden perspective: internationalisation inherently good, cooperative, intercultural Reality: many UK, US, Australian and other HEIs regard internationalisation as a mainly commercial activity creating markets, benefits and revenues Activities include offshore campuses, direct recruitment, degree franchising Measuring Internationalisation Uwe Brandenburg January

8 2. Reality and wishful thinking 8

9 Core assumptions and statements Internationalisation No goal in itself but effective instrument to achieve other goals can enhance education, research, civic engagement etc. Measuring Internationalisation So far too many assumptions made, to few things proved: we claim but we do not know! Satisfaction surveys dominate next to input measurement Outcome (impact) is not measured yet Measuring Internationalisation Uwe Brandenburg January

10 Modern Fairy Tale: The Emperor s new clothes but there is nothing behind it! Ah he is wearing the true values of internationalisation Exchange Adapted from: Measuring Internationalisation Uwe Brandenburg January

11 Short-phase mobility and cultural shock satisfaction high Medium crisis Culture shock adjustment rev. cult. shock mobility phases in Bachelor programmes shorten from 7-12 months to probably 3-5 months (currently 6 months) Thus: most students will return before the adjustment phase! low departure return timeline Measuring Internationalisation Uwe Brandenburg January

12 Recognition and pension funds 66% of EU students receive recognition of ECTS credits (Prime report) Statement of a representative of a known German university in an open seminar: you do not really mean that we should grant credits for all the things students put in their Learning Agreement, do you? Young researchers: Highly complex nature of pensions coupled with general lack of awareness makes it difficult for people to make informed decisions about mobility and pensions. (Ackers, 2008) Measuring Internationalisation Uwe Brandenburg January

13 Mobility is good in itself? Measuring Internationalisation Uwe Brandenburg January

14 Recruitment is good? international degree-seeking students in engineering in Germany double as likely not to graduate than domestic students IDP Australia: questionable provider standards in the private and the public sector; an overwhelming concentration of international students in business and management; the quality of students' English language skills; accusations of soft marking; and incidences of fraudulent behaviour by students and staff (see: Measuring Internationalisation Uwe Brandenburg January

15 More is better? Measuring Internationalisation Uwe Brandenburg January

16 Current belief in numbers What is the value of: Number of partnerships if: nothing happens? % of international students if: they do not feel integrated, are ill-prepared, fail,? % of outgoing students if: they come back during cultural shock, do not get recognition, do not acquire intercultural skills,? Claiming to be an international university if this is mainly lip service (mission statement) Measuring Internationalisation Uwe Brandenburg January

17 Therefore We need a new orientation towards: - Counting what counts - Looking at real Outcome and impact Measuring Internationalisation Uwe Brandenburg January

18 3. Measuring and assessing what really matters: memo 18

19 memo tool A SURVEY-BASED TOOL MEASURING THE EFFECTS OF INTERNATIONAL MOBILITY 50% Tool for universities and other HEIs PRE and POST survey for students Annual institutional reports, personal feedbacks for students Measuring Internationalisation Uwe Brandenburg January

20 What is memo tool? Tool for universities and other HEIs To assess study abroad programmes and measure real mobility outcomes that matter for employers Survey for INCOMING and OUTGOING students 2 SURVEYS - PRE-departure survey - POST-return survey Annual reports for universities, immediate feedback for students Data report and key findings in user-friendly format and charts Immediate personal feedback for students on psychometric part Measuring Internationalisation Uwe Brandenburg January

21 memo targets on what matters Employers assign higher importance to personality aspects than to field-specific knowledge and skills that gets quickly outdated today (Erasmus Impact Study, 2014) memo values were proved to be closely related to employability of alumni and their chances to get promoted to managerial positions Measuring Internationalisation Uwe Brandenburg January

22 Personality is not so easy to change CORE PERSONALITY: most difficult to develop BUT INTERNATION AL MOBILITY HAS THE POWER TO AFFECT IT SKILLS SELF- CONCEPT TRAIT, MOTIVE ATTITUDES, VALUES KNOWLEDGE SURFACE: most easily developed SOURCE: The Iceberg Model and Central and Surface Competencies, Spencer & Spencer, 1993 Measuring Internationalisation Uwe Brandenburg January

23 Why to measure mobility impact? Measuring Internationalisation Uwe Brandenburg January

24 Why to measure mobility impact? Students tend to overestimate mobility outcomes Not every student benefits from the exchange mobility Not all employers consider mobility to have an added value itself Measuring Internationalisation Uwe Brandenburg January

25 How it works? A survey-based tool grounded in the state-of-the-art research in psychometrics A questionnaire on students characteristics and perceptions Psychometric test assessing their personality traits related to employability and intercultural competence Data collected prior to and after the mobility to assess the added value Measuring Internationalisation Uwe Brandenburg January

26 From a student perspective... Measuring Internationalisation Uwe Brandenburg January

27 Student s personal feedback Measuring Internationalisation Uwe Brandenburg January

28 4. ERASMUS IMPACT STUDY 28

29 Erasmus Impact Study (EIS) Measuring Internationalisation Uwe Brandenburg January

30 Classical surveys have 4 deficiencies... Measuring Internationalisation Uwe Brandenburg January

31 EIS offered innovative solutions... Measuring Internationalisation Uwe Brandenburg January

32 EIS used 5 online surveys for 5 different target groups mobile and non-mobile students mobile and non-mobile alumni 652 employers (mainly SMEs) 964 higher education institutions individual responses in total the largest sample ever mobile and non-mobile staff (academic and nonacademic) Measuring Internationalisation Uwe Brandenburg January

33 Reasons to go abroad For more than 90% of students Opportunity to live abroad Student in Spain Learn / improve a foreign language Form new relationships Develop soft skills Improve career prospects Measuring Internationalisation Uwe Brandenburg January

34 Reasons NOT to go abroad For more than 50% of students Uncertainty about costs of study abroad Student in Spain Family reasons or personal relationships Lack of other financial resources needed to study abroad Measuring Internationalisation Uwe Brandenburg January

35 To employers, student personality matters! Measuring Internationalisation Uwe Brandenburg January

36 Not everybody is prepared to go abroad Difference in personality before going abroad: Memo total Memo factor Vigour Memo factor Tolerance of Ambiguity Memo factor Serenity Memo factor Decisiveness Memo factor Curiosity Memo factor Confidence 50.9% 53.6% 65.6% 69.0% 66.1% 68.9% 66.8% 70.5% 67.6% 69.9% 70.9% 71.6% 75.8% 75.5% Statistically significant difference in ex ante memo values between Erasmus (69.0%) and nonmobiles (65.6%) 50% 55% 60% 65% 70% 75% 80% Non-mobileErasmus Measuring Internationalisation Uwe Brandenburg January

37 Mobility changes 8-9 times more than life Memo factor values of students BEFORE and AFTER mobility: 72% Non-mobile students 0.3% change per year 71% 70% 69% 69.0% 70.4% 68% Erasmus students 67% 1.4% change in 6 months 66% 65% students ex ante students ex post Erasmus Non-mobile students Measuring Internationalisation Uwe Brandenburg January

38 Mobility pushes people beyond average After a stay abroad, Erasmus students had on average higher memo values than 70% of the students Measuring Internationalisation Uwe Brandenburg January

39 Not every students benefits 51% of all mobile students and 52% of Erasmus students improved their skills through a stay abroad not all students improve!?? Measuring Internationalisation Uwe Brandenburg January

40 Change of personality - perception vs. reality Students tend to overestimate their learning outcomes: 81% of students perceived an improvement in their personality traits while only 52% actually attained higher memo factors values Measuring Internationalisation Uwe Brandenburg January

41 Employment and career Erasmus alumni are almost half as likely to suffer from long-term unemployment (2% compared to 4% of nonmobiles) have more than 20% lower unemployment rate 5 yrs after graduation than non-mobiles are 44% more likely to have a managerial position than nonmobiles are twice as likely to hold jobs with international characteristics than nonmobiles Lower long-term unemployment Managerial positions International jobs Measuring Internationalisation Uwe Brandenburg January

42 Placements have especially strong impact Measuring Internationalisation Uwe Brandenburg January

43 Mobility influences the European attitude More than 80% of the Erasmus students felt a strong relationship to Europe felt more European after their stay abroad Measuring Internationalisation Uwe Brandenburg January

44 Mobility influences the place to live and work The percentage of mobile students that could imagine living abroad increased through stays abroad from 90% to 93% and for those who wanted to work abroad from 86% to 91% Measuring Internationalisation Uwe Brandenburg January

45 Mobility influences private relationships Life partner of different nationality 32% of mobile alumni and 33% of Erasmus alumni had a life partner of a different nationality. This was nearly three times more than among non-mobile alumni. Life partner met abroad 24% of mobile alumni and 27% of Erasmus alumni met their life partner while abroad. Measuring Internationalisation Uwe Brandenburg January

46 5. Employability Scan 46

47 How about measuring Employability Old approach Measuring short or mid-length competences and skills (e.g. IT) No lasting effects Easily changeable How to properly measure anything beyond classical skills? Risk of socially desired answer patterns Measuring Internationalisation Uwe Brandenburg January

48 How about measuring Employability Employability scan is better because A Student s personality is a key factor of readiness for success in a professional career and changes through studies but differently for different students while not every intervention has an impact and different students respond to interventions in different ways. Measuring Internationalisation Uwe Brandenburg January

49 How about measuring Employability Aims of the Employability scan Measure the starting level of students personality at the beginning of their studies. Measure regularly the development of personality traits of all students on their progress towards graduation. Describe the differences between subject cultures in different fields of studies. Identify risk groups to target support effectively. Measure the impact of individual interventions applied. Provide evidence for strategic decisions to adjust institutional policies in order to meet the needs of every students. Measuring Internationalisation Uwe Brandenburg January

50 How about measuring Employability Specifications online survey tool Each student surveyed once a year. Six memo factors related to employability: Confidence, Curiosity, Decisiveness, Problem-solving, Self-assessment and Tolerance. Every student receives personal feedback Additional questions included to allow for analysis by individual relevant groups and for evaluation of effects of individual institutional interventions Student characteristics: study programme, year of studies / enrolment cohort, family background, age, gender and nationality. Measuring Internationalisation Uwe Brandenburg January

51 How about measuring Employability Specifications Interventions a student might be subject of: international mobility and its type (study abroad, work placement, short term international mobility), aspects of internationalisation at home (foreign guest lectures, courses in foreign languages, interaction with incoming students, ), internships at home country, social engagement activities, specific curricular activities (such as summer courses, block semi-nars, soft skills trainings etc. to be further specified), political and social engagement (student unions and clubs, political participation), counselling and advisory services (to be further specified). Measuring Internationalisation Uwe Brandenburg January

52 How about measuring Employability Effect sizes of interventions by subject area intercultural training international lecturer Social engagement Internship Mobility Average Natural Sciences Sports Business/Economics Engineering Law Social Sciences Humanities Measuring Internationalisation Uwe Brandenburg January

53 5. Options for Israel 53

54 What Israel could do to improve true impact measurement of internationalisation Memo as a standard for all Israeli HEIs measuring the impact of mobility (outgoing and incoming): Regular tool financed by the Ministry Annual pre-to-post analysis Ministry receives report for overall Israeli situation Use it specifically for the Erasmus+ mobilities into and from Israel (comparison to overall European situation possible) Measuring Internationalisation Uwe Brandenburg January

55 What Israel could do to improve true impact measurement of internationalisation Employability scan Regular tool financed by the Ministry (and possibly employers) Annual assessment of all students in the HE system Possible to start with selected pilot subjects (e.g. Business, engineering and humanities) Building annually (full cycle after 3 years) Feedback document personalised and usable for job applications Ministry receives report on employabilty status of e.g. graduates per HEI, subject areas etc. Measuring Internationalisation Uwe Brandenburg January

56 What we could do for Israel Assessment of the Israeli mobility funding schemes (EIS+) Start as a one-off with opportunity to run regularly (e.g. every 3 years) Assessing all mobility schemes in comparison to Erasmus+ Opportunity to assess effectiveness and efficiency Measuring Internationalisation Uwe Brandenburg January

57 Thank you for your attention. Measuring Internationalisation Uwe Brandenburg January

58 For questions regarding memo and the employability scan Dana Petrova, CHE Consult GmbH (in English) 58