Meta-evaluation of the assessment system of learning outcomes and the development of the evaluation system in VET reform Hämeenlinna 29 November 2017 Counsellor of evaluation, PhD Mari Räkköläinen, FINEEC
ASSESSMENT OF LEARNING OUTCOMES in VET To indicate what the student knows and can do as a result of learning process, in relation to the educational goals and criteria 2
NATIONAL EVALUATION OF LEARNING OUTCOMES IN VET The aim of the evaluation of learning outcomes is to produce information on how well the students have achieved the objectives of National qualification requirements how well the students' competence meets the needs of working life The aim is also to develop and assure the quality of education and to support learning at local and national level 3
ASSESSMENT OF LEARNING OUTCOMES Since 2006 all initial vocational qualifications have included practical skills demonstrations for student competence assessment National evaluation of learning outcomes is based on the (local) skills demonstrations since 2007 (integrated system) No separate national tests AIM: Utility-focused and developmental oriented evaluation of learning outcomes 4
NATIONAL EVALUATION OF LEARNING OUTCOMES IN VOCATIONAL BASIC EDUCATION The qualification requirements for each vocational qualification steer the implementation and assessment of the demonstrations. Each year, evaluations of 2 4 qualification assessments have been started. All the providers offering the qualification in question and students are involved. Data will be collected over a period of three years. Learning outcome data has been supplemented with education providers self-assessments, assessment visits, demonstration plans and analyses of demonstration descriptions. The providers receive feedback on their results set in comparison with corresponding national comparison data. a national qualification-specific assessment report is published MR 27.11.2017 5
Characteristics of assessment of learning outcomes (in skills-demonstrations) Authentic assessment-> genuine, real, realistic Context-based assessment Competence-based assessment Criterion-referenced assessment The criteria of assessment are drawn from real working life requirements Assessing demonstrated competence -> the use and application of skills and knowledge Joint assessment (by teacher & student & working life representative) AIM: Assessment in itself produces learning and know-how Developmental assessment and inclusive QA-methods 28.11.2017 6
METHODOLOGICAL CHALLENGES caused by Characteristic of assessment of learning outcomes in vet Strengthening of competence-based and learning outcomes approach Integration of different purposes of evaluation and assessment (formative, summative, accountability) Quality assurance of assessment and evaluation processes Integration of formative and summative evaluation involves tensions that are interwoven with respect to confidence and quality assurance 7
DEVELOPMENT OF THE EVALUATION SYSTEM OF LEARNING OUTCOMES IN VET Meta-evaluation will be made to the evaluation system of learning outcomes (2016-2017)
The purpose of the meta-evaluation AIM is: To evaluate the effectiveness and utilization of the evaluation system, especially from the perspective of vet providers, the decision makers and the stakeholders To monitor the consequences of assessment and evaluation based on skills demonstrations, particularly the backwash to teaching and learning To produce information for development work of evaluation and assessment system and practices Development project of evaluation system will be launched based on the results of the meta-evaluation (2018) The requirements of the national reform of VET will be taken into account in development work 9
Evaluation questions 1. What kind of information has the evaluations of learning outcomes based on vocational skills demonstrations produced (2007 2017)? 2. How useful has the evaluation system for learning outcomes been for the beneficiaries and users? 3. What do the changes in the operating environment of vocational education, especially the reform and competencebased approach, mean for the learning outcome evaluation system? What requirements or challenges does the change bring to the evaluation of learning outcomes? 4. How should the learning outcome evaluation system be developed to ensure the effectiveness and confidence?
STAGES OF META-EVALUATION 2016-2017 Stage 1: SYNTHESIS OF LEARNING OUTCOMES 21 evaluation reports of the vet qualifications to analyse the learning outcomes ( grades, the quality of skills demonstrations, assessors, working life orientation ) to make synthesis of the recommendations Stage 2: QUESTIONNAIRES FOR VET PROVIDERS usefulness of evaluation system strengths and challenges ideas for future Stage 3: EXTERNAL EVALUATION external expert group interviews of key experts and decision makers evaluation panels and workshops (teachers, principles, students, working life) Stage 4: THE DEVELOPMENT PLAN AND THE PILOT OF NEW EVALUATION SYSTEM report and proposal for development process 11
Synthesis of evaluations (2007-2017) DATA Qualifications (evaluation reports Education fields Providers Students Skills-demonstrations Special need students sd:s Self-assessments of providers Descriptions of Skillsdemonstrations Skill-demonstration plans Assessment visits IN ALL 21 8 99 21 183 97 454 7 595 493 67 122 395 35 MR 27.11.2017
Vocational competence and work orientation Students vocational skills and competences excellent (48 %) or good (45 %), satisfactory (7%) weakest skills in theoretical knowledge best performance in key competences in general women get better grades Work orientation varies a lot between qualifications and education providers skills demonstrations at work places (53 %), at school (45 %) joint assessment 57 % (teacher & student & work instructors) on average 25 % of work instructors trained for assessing and supervising competent teachers 86 %, on average 29 % of teachers have had in-service-training at work places or enterprises Mari Räkköläinen 27.11.2017
Effectiveness and utilization of the evaluations At national level systematic, regular and established practice Gives more useful information about quality of the student assessment process and implementation of assessment practices than about learning outcomes There is great variation in the utilization of the evaluation results (providers, educational administration, MoE, FNBE, working life) In general evaluation system is considered to be important and useful, but the problem is usability and comparability Skills demonstrations has improved learning and increased validity of assessment of learning outcomes 14
RESULTS OF SURVEY Factors that facilitated or prevented utilisation of data obtained from the evaluations FACTORS THAT FACTORS THAT FACILITATED UTILISATION OF DATA OBTAINED FROM THE EVALUATIONS FACILITATED UTILISATION OF DATA OBTAINED FROM THE EVALUATIONS systematicity utilisation of the data is not organised systematically, it is not the responsibility of anyone functional quality assurance system lack of quality awareness in the educational institution, development philosophy is out of date reliability of results weakness of reliability of results availability of data, (particularly comparative data and information on development needs) clear mode of presenting the results takes too long to obtain the results challenge in finding or interpreting the relevant data data is too general data is not appropriate -> cannot be utilised in development joint processing of data with various actors weakness of communication concerning results positive attitude negative attitude development of own competence during the process lack of competence lack of time organisational changes MR 27.11.2017 15
RESULTS OF SURVEY Summary What is positive? What should be improved? National coverage Feedback for educational institutions on their own results Obtaining comparative data at national level Self-assessments of providers and assessment visits Participation (eg. Appointment of contact persons by the education provider) Activity (eg. Vocational qualification expert included in the evaluation) Deepen and lightening Utilisation of electronic systems Increase in qualitative feedback Less but more focused information and feedback Taking into account competence-based approach, reform and future changes in VET Mari Räkköläinen 27.11.2017 16
NEXT STEPS Proposal and plan for a new evaluation system Piloting as part of VET reform HIGHLIGHTING BENEFICIARIES PERSPECTIVE IN ORDER TO DEVELOP THE SYSTEM The evaluation process and the uses of results has to be linked with development work and the reform (at all levels) In context-based evaluation validity has been considered more significant than over all generalization (comparability) due to the benefits accused at local level (schools, learning methods, teachers, workplaces), but there is a need for quality assurance Indicators describing the level of work orientation (relevance to work) are required Indicators for strengthening students viewpoint
Discussion: Is contextual and processual assessment of learning outcomes possible? How choices made during the learning and instruction processes and pedagogy (work relation and orientation) link with student competencies? Are efforts to increase awareness of critical assessment factors among education providers, teachers, representatives of working life and students even more useful than to achieve the highest degree of measurability and reliability? Should the quality of skills demonstrations be higher in order to establish an evaluation system based on quality assurance? MR 27.11.2017 Mari Räkköläinen 18
EVALUATION TENSIONS (Räkköläinen 2011) INTERNAL private, personal, local EXTERNAL public general FORMATIVE validity authenticity SUMMATIVE reliability, comparability TRUST local autonomy, professional skills CONTROL central guidance standards
Discussion: How to increase the impact of evaluations?
PERSPECTIVE PROACTIVE STRATEGIC REFORMATIVE FUTURE and CHANGE 28.11.2017 21
QUALITY IN EVALUATION: Impact CONFIDENCE Openness SIGNIFICANCE Usability EFFECTIVINESS Change AIM: Intelligent accountability Participants to be professionally responsible and to be active participants in the evaluation process and interpretation of evaluation results MR 24.11.2017 22
Sources Chelimsky, E. & Shadish, W.R. 1997. Evaluation for the 21st century: A handbook. Thousand Oaks, Ca: Fetterman, D. 2001. Foundations of empowerment evaluation. London: Sage Patton, M.Q. 1997. Utilization Focused Evaluation. The New Century Text. London: Sage Publications. Räkköläinen, M. 2011. What do skills demonstrations reveal? The reliability and confidence in the assessment process of skills demonstrations in VET]. Acta Universitatis Tamperensis 1636. Tampere University Press: Tampere. (doctoral dissertation) Räkköläinen, M & Räisänen, A. 2013. Assessment of Learning Outcomes in Finnish vocational education and training. Assessment in Education: Principles, Policy & Practice (2013). Assessment of learning outcomes in Finnish vocational education and training, Assessment in Education: Principles, Policy & Practice, DOI: 10.1080/0969594X.2013.838938. To link to this article: http://dx.doi.org/10.1080/0969594x.2013.838938 Räisänen, A. & Räkköläinen, M. 2014. Developmental Assessment of Learning Outcomes. In S. Kalliola (ed.) Evaluation as a Tool for Research, Learning and Making Things Better. Cambridge Scholars Publishing. 23
VISION Evaluation is about development and future, not only past and judgement 24
We would be pleased to provide more information on evaluation www.osaamisenarviointi.karvi.fi Mari Räkköläinen Arviointineuvos, projektipäällikkö puh. 029 533 5527 mari.rakkolainen@karvi.fi Jukka Jalolahti Arviointisuunnittelija puh. 029 533 5516 jukka.ruuskanen@karvi.fi Veera Stylman Arviointisuunnittelija puh. 029 533 5555 veera.stylman@karvi.fi Henna Heinäjärvi Harjoittelija puh. 029 533 5562 henna.heinajarvi@karvi.fi 27.11.2017 25