Measuring what really matters ways of impact measurement of employability and the effects of mobility Uwe Brandenburg CHE Consult GmbH January 2016
Connection to IRIS project 2
IRIS project Improving internationalisation of Israeli colleges Role of CHE Consult Cross-checking status quo assessment Developing indicators for internationalisation Creating a measuring system Producing individual roadmaps for the colleges to implement internationalisation procedures and structures Measuring Internationalisation Uwe Brandenburg January 2016 3
1. Background 4
Internationalisation definition of term itself and application changed considerably Today, standard definition used: Internationalisation at the national/sector/ institutional levels is defined as the process of integrating an international, intercultural or global dimension into the purpose, functions or delivery of higher education. (Knight 2009, 7) Measuring Internationalisation Uwe Brandenburg January 2016 5
Internationalisation buzzword of at least the last decade and a half started off in political science and governmental relations, entered higher education in 1980s last two decades: concept of the internationalisation moved from fringe to core of institutional agenda Effects: tends to become a conditio sine qua non creates an atmosphere of high risk of lip service and give to the emperor Measuring Internationalisation Uwe Brandenburg January 2016 6
Internationalisation Key terms: Process incorporating global aspects into HE Differentiation between internationality and internationalisation (Brandenburg & Federkeil 2007) Value-laden perspective: internationalisation inherently good, cooperative, intercultural Reality: many UK, US, Australian and other HEIs regard internationalisation as a mainly commercial activity creating markets, benefits and revenues Activities include offshore campuses, direct recruitment, degree franchising Measuring Internationalisation Uwe Brandenburg January 2016 7
2. Reality and wishful thinking 8
Core assumptions and statements Internationalisation No goal in itself but effective instrument to achieve other goals can enhance education, research, civic engagement etc. Measuring Internationalisation So far too many assumptions made, to few things proved: we claim but we do not know! Satisfaction surveys dominate next to input measurement Outcome (impact) is not measured yet Measuring Internationalisation Uwe Brandenburg January 2016 9
Modern Fairy Tale: The Emperor s new clothes but there is nothing behind it! Ah he is wearing the true values of internationalisation Exchange Adapted from: http://de.toonpool.com/user/43/files/des_kaisers_neue_kleider_262355.jpg Measuring Internationalisation Uwe Brandenburg January 2016 10
Short-phase mobility and cultural shock satisfaction high Medium crisis Culture shock adjustment rev. cult. shock mobility phases in Bachelor programmes shorten from 7-12 months to probably 3-5 months (currently 6 months) Thus: most students will return before the adjustment phase! low departure return timeline Measuring Internationalisation Uwe Brandenburg January 2016 11
Recognition and pension funds 66% of EU students receive recognition of ECTS credits (Prime report) Statement of a representative of a known German university in an open seminar: you do not really mean that we should grant credits for all the things students put in their Learning Agreement, do you? Young researchers: Highly complex nature of pensions coupled with general lack of awareness makes it difficult for people to make informed decisions about mobility and pensions. (Ackers, 2008) Measuring Internationalisation Uwe Brandenburg January 2016 12
Mobility is good in itself? Measuring Internationalisation Uwe Brandenburg January 2016 13
Recruitment is good? international degree-seeking students in engineering in Germany double as likely not to graduate than domestic students IDP Australia: questionable provider standards in the private and the public sector; an overwhelming concentration of international students in business and management; the quality of students' English language skills; accusations of soft marking; and incidences of fraudulent behaviour by students and staff (see: http://www.theaustralian.com.au/higher-education/slump-in-chinesestudents-begins/story-e6frgcjx-1226049380064) Measuring Internationalisation Uwe Brandenburg January 2016 14
More is better? Measuring Internationalisation Uwe Brandenburg January 2016 15
Current belief in numbers What is the value of: Number of partnerships if: nothing happens? % of international students if: they do not feel integrated, are ill-prepared, fail,? % of outgoing students if: they come back during cultural shock, do not get recognition, do not acquire intercultural skills,? Claiming to be an international university if this is mainly lip service (mission statement) Measuring Internationalisation Uwe Brandenburg January 2016 16
Therefore We need a new orientation towards: - Counting what counts - Looking at real Outcome and impact Measuring Internationalisation Uwe Brandenburg January 2016 17
3. Measuring and assessing what really matters: memo 18
memo tool A SURVEY-BASED TOOL MEASURING THE EFFECTS OF INTERNATIONAL MOBILITY 50% Tool for universities and other HEIs PRE and POST survey for students Annual institutional reports, personal feedbacks for students Measuring Internationalisation Uwe Brandenburg January 2016 19
What is memo tool? Tool for universities and other HEIs To assess study abroad programmes and measure real mobility outcomes that matter for employers Survey for INCOMING and OUTGOING students 2 SURVEYS - PRE-departure survey - POST-return survey Annual reports for universities, immediate feedback for students Data report and key findings in user-friendly format and charts Immediate personal feedback for students on psychometric part Measuring Internationalisation Uwe Brandenburg January 2016 20
memo targets on what matters Employers assign higher importance to personality aspects than to field-specific knowledge and skills that gets quickly outdated today (Erasmus Impact Study, 2014) memo values were proved to be closely related to employability of alumni and their chances to get promoted to managerial positions Measuring Internationalisation Uwe Brandenburg January 2016 21
Personality is not so easy to change CORE PERSONALITY: most difficult to develop BUT INTERNATION AL MOBILITY HAS THE POWER TO AFFECT IT SKILLS SELF- CONCEPT TRAIT, MOTIVE ATTITUDES, VALUES KNOWLEDGE SURFACE: most easily developed SOURCE: The Iceberg Model and Central and Surface Competencies, Spencer & Spencer, 1993 Measuring Internationalisation Uwe Brandenburg January 2016 22
Why to measure mobility impact? Measuring Internationalisation Uwe Brandenburg January 2016 23
Why to measure mobility impact? Students tend to overestimate mobility outcomes Not every student benefits from the exchange mobility Not all employers consider mobility to have an added value itself Measuring Internationalisation Uwe Brandenburg January 2016 24
How it works? A survey-based tool grounded in the state-of-the-art research in psychometrics A questionnaire on students characteristics and perceptions Psychometric test assessing their personality traits related to employability and intercultural competence Data collected prior to and after the mobility to assess the added value Measuring Internationalisation Uwe Brandenburg January 2016 25
From a student perspective... Measuring Internationalisation Uwe Brandenburg January 2016 26
Student s personal feedback Measuring Internationalisation Uwe Brandenburg January 2016 27
4. ERASMUS IMPACT STUDY 28
Erasmus Impact Study (EIS) Measuring Internationalisation Uwe Brandenburg January 2016 29
Classical surveys have 4 deficiencies... Measuring Internationalisation Uwe Brandenburg January 2016 30
EIS offered innovative solutions... Measuring Internationalisation Uwe Brandenburg January 2016 31
EIS used 5 online surveys for 5 different target groups... 56 733 mobile and non-mobile students 18 618 mobile and non-mobile alumni 652 employers (mainly SMEs) 964 higher education institutions 78 891 individual responses in total the largest sample ever 4 986 mobile and non-mobile staff (academic and nonacademic) Measuring Internationalisation Uwe Brandenburg January 2016 32
Reasons to go abroad For more than 90% of students Opportunity to live abroad Student in Spain Learn / improve a foreign language Form new relationships Develop soft skills Improve career prospects Measuring Internationalisation Uwe Brandenburg January 2016 33
Reasons NOT to go abroad For more than 50% of students Uncertainty about costs of study abroad Student in Spain Family reasons or personal relationships Lack of other financial resources needed to study abroad Measuring Internationalisation Uwe Brandenburg January 2016 34
To employers, student personality matters! Measuring Internationalisation Uwe Brandenburg January 2016 35
Not everybody is prepared to go abroad Difference in personality before going abroad: Memo total Memo factor Vigour Memo factor Tolerance of Ambiguity Memo factor Serenity Memo factor Decisiveness Memo factor Curiosity Memo factor Confidence 50.9% 53.6% 65.6% 69.0% 66.1% 68.9% 66.8% 70.5% 67.6% 69.9% 70.9% 71.6% 75.8% 75.5% Statistically significant difference in ex ante memo values between Erasmus (69.0%) and nonmobiles (65.6%) 50% 55% 60% 65% 70% 75% 80% Non-mobileErasmus Measuring Internationalisation Uwe Brandenburg January 2016 36
Mobility changes 8-9 times more than life Memo factor values of students BEFORE and AFTER mobility: 72% Non-mobile students 0.3% change per year 71% 70% 69% 69.0% 70.4% 68% Erasmus students 67% 1.4% change in 6 months 66% 65% students ex ante students ex post Erasmus Non-mobile students Measuring Internationalisation Uwe Brandenburg January 2016 37
Mobility pushes people beyond average After a stay abroad, Erasmus students had on average higher memo values than 70% of the students Measuring Internationalisation Uwe Brandenburg January 2016 38
Not every students benefits 51% of all mobile students and 52% of Erasmus students improved their skills through a stay abroad not all students improve!?? Measuring Internationalisation Uwe Brandenburg January 2016 39
Change of personality - perception vs. reality Students tend to overestimate their learning outcomes: 81% of students perceived an improvement in their personality traits while only 52% actually attained higher memo factors values Measuring Internationalisation Uwe Brandenburg January 2016 40
Employment and career Erasmus alumni are almost half as likely to suffer from long-term unemployment (2% compared to 4% of nonmobiles) have more than 20% lower unemployment rate 5 yrs after graduation than non-mobiles are 44% more likely to have a managerial position than nonmobiles are twice as likely to hold jobs with international characteristics than nonmobiles Lower long-term unemployment Managerial positions International jobs Measuring Internationalisation Uwe Brandenburg January 2016 41
Placements have especially strong impact Measuring Internationalisation Uwe Brandenburg January 2016 42
Mobility influences the European attitude More than 80% of the Erasmus students felt a strong relationship to Europe felt more European after their stay abroad Measuring Internationalisation Uwe Brandenburg January 2016 43
Mobility influences the place to live and work The percentage of mobile students that could imagine living abroad increased through stays abroad from 90% to 93% and for those who wanted to work abroad from 86% to 91% Measuring Internationalisation Uwe Brandenburg January 2016 44
Mobility influences private relationships Life partner of different nationality 32% of mobile alumni and 33% of Erasmus alumni had a life partner of a different nationality. This was nearly three times more than among non-mobile alumni. Life partner met abroad 24% of mobile alumni and 27% of Erasmus alumni met their life partner while abroad. Measuring Internationalisation Uwe Brandenburg January 2016 45
5. Employability Scan 46
How about measuring Employability Old approach Measuring short or mid-length competences and skills (e.g. IT) No lasting effects Easily changeable How to properly measure anything beyond classical skills? Risk of socially desired answer patterns Measuring Internationalisation Uwe Brandenburg January 2016 47
How about measuring Employability Employability scan is better because A Student s personality is a key factor of readiness for success in a professional career and changes through studies but differently for different students while not every intervention has an impact and different students respond to interventions in different ways. Measuring Internationalisation Uwe Brandenburg January 2016 48
How about measuring Employability Aims of the Employability scan Measure the starting level of students personality at the beginning of their studies. Measure regularly the development of personality traits of all students on their progress towards graduation. Describe the differences between subject cultures in different fields of studies. Identify risk groups to target support effectively. Measure the impact of individual interventions applied. Provide evidence for strategic decisions to adjust institutional policies in order to meet the needs of every students. Measuring Internationalisation Uwe Brandenburg January 2016 49
How about measuring Employability Specifications online survey tool Each student surveyed once a year. Six memo factors related to employability: Confidence, Curiosity, Decisiveness, Problem-solving, Self-assessment and Tolerance. Every student receives personal feedback Additional questions included to allow for analysis by individual relevant groups and for evaluation of effects of individual institutional interventions Student characteristics: study programme, year of studies / enrolment cohort, family background, age, gender and nationality. Measuring Internationalisation Uwe Brandenburg January 2016 50
How about measuring Employability Specifications Interventions a student might be subject of: international mobility and its type (study abroad, work placement, short term international mobility), aspects of internationalisation at home (foreign guest lectures, courses in foreign languages, interaction with incoming students, ), internships at home country, social engagement activities, specific curricular activities (such as summer courses, block semi-nars, soft skills trainings etc. to be further specified), political and social engagement (student unions and clubs, political participation), counselling and advisory services (to be further specified). Measuring Internationalisation Uwe Brandenburg January 2016 51
How about measuring Employability Effect sizes of interventions by subject area intercultural training international lecturer Social engagement Internship Mobility 0.00 0.10 0.20 0.30 0.40 0.50 0.60 Average Natural Sciences Sports Business/Economics Engineering Law Social Sciences Humanities Measuring Internationalisation Uwe Brandenburg January 2016 52
5. Options for Israel 53
What Israel could do to improve true impact measurement of internationalisation Memo as a standard for all Israeli HEIs measuring the impact of mobility (outgoing and incoming): Regular tool financed by the Ministry Annual pre-to-post analysis Ministry receives report for overall Israeli situation Use it specifically for the Erasmus+ mobilities into and from Israel (comparison to overall European situation possible) Measuring Internationalisation Uwe Brandenburg January 2016 54
What Israel could do to improve true impact measurement of internationalisation Employability scan Regular tool financed by the Ministry (and possibly employers) Annual assessment of all students in the HE system Possible to start with selected pilot subjects (e.g. Business, engineering and humanities) Building annually (full cycle after 3 years) Feedback document personalised and usable for job applications Ministry receives report on employabilty status of e.g. graduates per HEI, subject areas etc. Measuring Internationalisation Uwe Brandenburg January 2016 55
What we could do for Israel Assessment of the Israeli mobility funding schemes (EIS+) Start as a one-off with opportunity to run regularly (e.g. every 3 years) Assessing all mobility schemes in comparison to Erasmus+ Opportunity to assess effectiveness and efficiency Measuring Internationalisation Uwe Brandenburg January 2016 56
Thank you for your attention. Measuring Internationalisation Uwe Brandenburg January 2016 57
For questions regarding memo and the employability scan Dana Petrova, CHE Consult GmbH E-Mail: dana.petrova@che-consult.de www.memo-tool.net (in English) www.che-consult.de 58