AN INTERNATIONAL APPROACH TO ASSESSING HIGHER EDUCATION LEARNING OUTCOMES: THE OECD/CAE INITIATIVE Dirk Van Damme OECD/EDU/SBS
THE CASE FOR TRANSPARENCY ON LEARNING OUTCOMES 1. VALUE FOR MONEY 2
Financial inputs in higher education increasing Total (public & private) financial investment grew Between 2008 and 2014 on average across OECD increase of >5% in per student expenditure With huge differences between countries, increases higher in countries with below-average expenditure, catching up Yearly per student expenditure is now 15 KUS$ Total expenditure increased from 1.3% GDP in 2000 to 1.6% GDP in 2014 3
Financial inputs in higher education increasing Private expenditure has increased a lot 32% of total expenditure (0.5% GDP) comes from private sources, mainly tuition fees Increase from 25% in 2000 Total private expenditure increased with 32% since 2005 >50% in Israel, US, Australia, Japan, UK, Korea and Chile 4
The value-for-money argument Efficiency and value-for-money become very important policy considerations Both for governments and students/families Cost of higher education becoming political issue in many countries What are students actually buying? Very weak relationship between cost and actual product, benefits and outcomes Value-for-money depends enormously on institution and field of study 5
Value for money? 6
THE CASE FOR TRANSPARENCY ON LEARNING OUTCOMES 2. QUALITY OF LEARNING 7
95th percentile mean score tertiary 25-34y Finland Czech Republic Flanders (Belgium) Netherlands Austria Sweden Japan Germany Slovak Republic Norway Denmark France Average Estonia Australia Northern Ireland (UK) United States Poland Canada Korea Italy Ireland England/N. Ireland (UK) England (UK) Spain 200 220 240 260 280 300 320 340 360 380 Numeracy scores of tertiary educated adults of 25-34y old 8
Literacy equivalent of tertiary qualifications 9
Japan Finland Netherlands Sweden Australia Norway Flanders (Belgium) England (UK) England/N. Ireland (UK) United States Czech Republic OECD average Poland Canada Northern Ireland (UK) Austria Germany Ireland France Denmark Estonia Slovak Republic Korea Russian Federation Spain Italy Numeracy equivalent of tertiary qualifications Proportion of 25-64 year-olds scoring at PIAAC numeracy level 4 and 5, by educational attainment of the population (2012) Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education 40 35 30 25 20 15 10 5 0 10
THE CASE FOR TRANSPARENCY ON LEARNING OUTCOMES 3. FROM CREDENTIALS TO COMPETENCES 11
Concerns about quality and value of qualifications Concerns about the quality and added-value of a university experience Academically Adrift: limited improvement in academic skills What is the relative contribution of selection versus teaching and learning in the production of high-quality graduates; what is the actual learning gain Doubts on the quality of the teaching and learning experience at universities 12
Concerns about over-qualification 13
Concerns about quality and value of qualifications Concerns about the quality and added-value of a university experience Academically Adrift: limited improvement in academic skills What is the relative contribution of selection versus teaching and learning in the production of high-quality graduates; what is the actual learning gain Doubts on the quality of the teaching and learning experience at universities 14
Formal qualifications vs skills 15
Erosion of degrees? 16
UNDERSTANDING (LACK OF) TRANSPARENCY IN HIGHER EDUCATION: A PROBLEM OF INFORMATION ASYMMETRY 17
Information asymmetry What do we know about how quality of teaching and learning results in high-quality output, and socially interesting outcomes? Information asymmetry: both public and private actors (and financers) of higher education have very little understanding of what they actually are spending money for Increase of investment has not been accompanied by an empowerment of the input side to make smart choices through better information In a diversifying system what matters is the output: what have students learned? 18
Transparency How to improve transparency? Performance management systems: essentially bureaucratic control systems, very rarely comparative and tuned to better decision-making Quality assurance arrangements: increasingly relativistic ( fitness-for-purpose ), bureaucratic and inward oriented (internal QA), unfit for external transparency, nor empowering students Student satisfaction surveys: empowering, but perception-based, largely unrelated to teaching and learning quality 19
Transparency How to improve transparency? Input measures as proxies for teaching and learning quality: above a threshold level input has a weak relationship to quality (cfr PISA) Research measurement and bibliometric indicators as proxies for teaching and learning quality: most inaccurate, unfair to wide range of institutions and institutional diversity Reputations: act as (too?) powerful tools, also in rankings, but inaccurate, often outdated, inimical to newcomers and innovators 20
Transparency requires metrics on learning gain Sound metrics of learning are very much needed To reassure governments and families about the value-for-money of investments To reward institutions who invest in improving teaching and learning and are not compensated through other measures To value institutional diversification To reward and foster quality improvement through mutual learning To compensate for the over-reliance of rankings on research and reputation metrics 21
AHELO ( 2008-2015) 22
Assessing Higher Education Learning Outcomes (AHELO) Feasibility Study (2008-12) Objective: test whether it is possible to For measure what undergraduate students know and can do upon graduation across diverse countries, languages, cultures and types of institutions Generic skills Economics Engineering 23
A range of geographic, linguistic and cultural backgrounds involved Generic Skills Colombia Egypt Finland Korea Kuwait Mexico Norway Slovak Republic United States (CT, MO, PA) Engineering Abu Dhabi Australia Canada (Ontario) Colombia Egypt Japan Mexico Russian Fed. Slovak Republic Observers Bahrein Brazil Saudi Arabia Singapore Economics Belgium (Fl.) Egypt Italy Mexico Netherlands Russian Fed. Slovak Republic 24
Quick facts on AHELO Feasibility Study Data collection from February to June 2012 17 countries involved in 25 strand replications Data collected from over 23,000 students 4,900 faculties 270 institution coordinators 1,000 test sessions and 20,000 computers involved Scoring completed in June 2012 Analysis of results and findings Final reporting 2012-13 25
The collapse of AHELO After several rounds of discussion the EDPC decided NOT to pursue the proposed AHELO Main Study Important differences of views between countries Organised reaction from some higher education networks Active opposition from some major countries 26
ASSESSING STUDENTS LEARNING OUTCOMES RECENT DEVELOPMENTS 27
Recent developments Comparative assessment of learning outcomes of graduates is the most promising approach to measure teaching and learning excellence OECD s AHELO project National research projects in Germany, UK, Italy CLA and various other initiatives in US European Commission supporting CALOHEE project in Tuning framework 28
The OECD-CAE initiative to implement CLA+ in countries Partnership based on MOU between OECD and CAE Focus on generic academic skills, such as critical thinking, scientific reasoning, academic writing, etc. Performance Task (PT) and set of document-based Selected-Response Questions (SRQs) Countries (different stages of implementation): Italy, Germany (KoKoHs), England (TEF), Ontario, Finland, Japan, Hong Kong, Mexico, Peru, Chile, Russia, 29
The OECD-CAE initiative to implement CLA+ in countries Methodology of the CLA+ instrument: Focus on generic academic skills, assessing students ability to reason scientifically and quantitatively, to critically read and evaluate information, and to critique arguments Performance Task (PT) and Selected-Response Questions (SRQ) Three scales: Analysis and Problem-Solving, Writing Effectiveness, and Writing Mechanics Students can acquire certificates or badges High validity and reliability of the instrument 30
Summarizing: assessment of students learning outcomes is needed in order To improve our understanding of what students actually learn in higher education To exchange reputations with empirically grounded observations of quality of teaching & learning To gradually transform the field on which credentials are traded into a more level playing field To provide better information to students and employers about the quality of teaching & learning experiences To develop feedback loops to improve teaching and learning To reward and incentivise institutions that significantly improve their teaching & learning environments To re-confirm the value of teaching as part of the university s mission next to research 31
Thank you! dirk.vandamme@oecd.org www.oecd.org/edu/ceri twitter @VanDammeEDU 32