Applied Physics department Groene Loper, Eindhoven P.O. Box 513, 5600 MB Eindhoven The Netherlands www.tue.nl Authors dr. S.M. Gomez Puente Education Policy & Quality Assurance Advisor (AP) Prof. dr. K.A.H. van Leeuwen AP Program director Date 18 June 2017 Version 1 Quality assurance system: A plan to monitor, evaluate and improve the quality of education Applied Physics department
Table of contents Title Quality assurance 1 Introduction...3 2 The quality assurance systems at the Applied Physics department: General description...4 2.1 The monitoring of the quality of education: the process and procedures....4 2.2 Support to faculty staff on improvement of quality of education.. 7 3 Criteria and frequency for the monitoring of the quality of education..... 8 4 Instruments to monitor and evaluate the quality of education at course and program level. 9
1 Introduction This document 1 provides an overview of the quality assurance (QA) system of the Applied Physics (AP) department. The quality assurance system, i.e., monitoring, evaluation and improvement plans, comprises both the quality assurance cycle at course level, the so-called kleine kwaliteitszorg, as well as the quality assurance cycle at program level. The relevance of the monitoring and the evaluation of the quality of education is essential towards becoming accountable to both internal (AP internal forums such as the AP Departmental Board, Program Committee (PC), Examination Committee (EC)), as well as external bodies (i.e., TU/e Executive Board, CCKO 2, Supervisory Board, NVAO 3, etc.) upon which the AP educational management is responsible to. In addition, the monitoring and evaluation of the quality of education is fundamental towards meeting the NVAO accreditation standards. In the following sections, the AP quality assurance system is described including a general overview of the process of safeguarding and controlling the quality of education, the process and the procedures. Moreover, we also provide a description of the primary stakeholders involved in this process and the instruments applied throughout the quality assurance system. 1 The structure and the different quality assurance elements included in this report follow in general the structure of the Bachelor College (BC) quality assurance system. 2 Centrale Commissie Kwaliteitszorg Onderwijs (CCKO) Central Committee for Quality Care in Education 3 Nederlands-Vlaamse Accreditatieorganisatie (NVAO) The Accreditation Organization of the Netherlands and Flanders 3 Quality assurance system Applied Physics department /June, 2017
2 The quality assurance system at the Applied Physics department: General description The Applied Physics department is committed to the NVAO and TU/e standards regarding the quality of education. The method that frames the mechanism to assure the quality of education is the Plan- Do-Check-Act (PDCA) cycle (see Figure 1). The PDCA method aims at applying an iterative procedure that allows for continuous identification of actions for improvement. The method consists of several steps, i.e., evaluation of education, decision on changes that need to be made, planning and carrying out the improvements, and finally, monitoring and evaluating the planned actions against expected results. The implementation of this method includes, therefore, regular analyses conducted on all educational aspects and it cuts across all levels (i.e., courses and study program). Figure 1. PDCA method 2.1 The monitoring of the quality of education: the process and procedures To assure quality of education, the AP educational management department (OITN 4 ) monitors and evaluates the process and the products regarding courses and study program (See Table 1). Table 1. Overview of criteria applied to monitor and evaluate education at course and at program level Course level Process criteria Education (set-up & organization)/prior knowledge Lectures/Instructions/Supervised self-study/projects Lecturers Assessment Study material Workload Value and pleasure with the course/project Course and program level Product criteria Pass rates course level Learning yield/pass rates cohort level Analyses of curriculum changes (i.e., Maths disciplinary line; interim tests, etc.) Quality and satisfaction on study program (NSE, Monitoring survey) Transfer bachelor and master curriculum surveys, etc. 4 Opleidingsinstituut Technische Natuurkunde (OITN) 4 Quality assurance system Applied Physics department /June, 2017
Regarding course level, the AP educational department follows the Bachelor College (BC) standard indicators for the review of the course evaluations 5 (See Table 2). The standard indicators are: Table 2. Bachelor College standard indicators* Meaning Course rating Evaluation scores on items Insufficient < 5,5 < 3 Moderate 5,5 6 3 3,2 Sufficient 6 8 3,2 4 Excellent 8 4 *These indicators are taken as a reference framework to decide to take actions by the AP education policy and quality assurance advisor. However, the decision to take actions is not based only the result of one item score. More issues are taken into consideration to, for instance, hold a meeting with the lecturer(s). Moreover, the information gathered from teachers and students councils (i.e., p-/pp-/m-raad) are also sources of information used to get an insight of the quality of courses as well. This information is also used during the meetings with the faculty staff. These meetings serve as a feedback mechanism in order to improve the educational elements in the courses which still need some attention. In some cases, a follow-up students group (college volggroep) provides also feedback to the lecturers. These groups are requested by the AP education policy and quality assurance advisor in case the courses need to be monitored. The groups are organized by STOOR 6. (See Figure 2.Course evaluation flow). Exceptionally, monitoring systems are created to follow-up specific themes. This is the case, for instance, of the Professional Skills after the BC audit has been carried out; or the introduction of the English bachelor program. With regards to the later, OITN will organize review meetings with first year students together with representatives of vd Waals and STOOR, with the purpose of gaining an overview on how the transition to an English bachelor program has taken place. Actions for adjustments will follow if needed. Regarding the quality of exams and according to the AP assessment policy 7, courses with pass rates below 60% and above 90% are subject to further research. Following this norm of course pass rates, the AP educational policy advisor carries out statistical test analysis. The ECTN Quality of exams Committee, i.e., Borgingscommissie, supported by the AP educational policy and quality assurance advisor is in charge of controlling the quality of exams. This committee revises the exams and hold a meeting with the responsible teacher(s) of the course to further analyze, from a quality assessment point of view, the possible causes of course results. In addition, an improvement plan regarding examination is also discussed. Furthermore, the AP Examination Committee (ECTN) has developed an assessment procedure to monitor and control the quality of exams based on the assessment indicators for the construction of exams, i.e., following transparency, reliability and validity 8. Among others, the procedure includes that the final test is developed by the lecturer(s) of the course and is reviewed by other lecturer(s); the objectives/contents of the course are used as the starting point for the construction of the test; the final test includes an overview of the points that can be obtained per sub-question and an answer model. (For further information on this topic, see section The process and procedures for quality assurance of exams, in AP assessment policy). The monitoring and quality control on course and program level is communicated with both internal and external stakeholders and takes place on regular basis. The management reports on pass rates, the 5 The course evaluations are carried out with the TU/e EvaSys survey management system. The TU/e quality assurance and evaluation department at central level organizes the process of making the questionnaires, sending these out to the students, and processing the information which is sent afterwards to the education policy and quality assurance advisor in each department. 6 STudenten Onderwijs ORganisatie. STOOR. 7 Facultair Toetsbeleid Technische Natuurkunde (2015). 8 Van Berkel, H., Bax A. (2014). Toetsen in het hoger onderwijs. Bohn Stafleu van Loghum. Houten. 5 Quality assurance system Applied Physics department /June, 2017
students questionnaires and the improvement plans are reviewed and discussed both during the Program Committee and the Examination Committee meetings once per quarter. In addition, pass rates are also presented during the bachelor and master students council meetings once every quarter. Students filling in the course questionnaires get also an overview of the item scores of the course evaluation. According to the AP departmental policy, the summary of the meetings with the lecturers, together with the students questionnaires are sent to the chair of the research group so that these documents can be discussed during the annual evaluation meeting with each concerned lecturer. Figure 2. Course evaluation flow 9 9 Adapted from Toekomstige Onderwijsorganisatie (TOO) model on Course evaluation flow. 6 Quality assurance system Applied Physics department /June, 2017
At program level, the AP educational management department monitors and evaluates continuously all educational elements of the study program (See Table 1 Overview of criteria applied to monitor and evaluate education at course and at program level). These are, for instance, analysis of educational policy regulations (i.e., BSA; effects of maths disciplinary curriculum line; PRV curriculum; effects of interim test, etc.). A progress overview on the study program is provided once a year to the PC in the form of BSc and MSc annual reports. In addition, specific information on, for instance, students progress on first year math results, the duration of bachelor end project (BEP) or master internship and graduation projects, among other issues upon request, are also reviewed frequently. Twice a year monitoring meetings take place with the Deans of the BC and GS to follow up educational issues. Also, the AP Departmental Board meets regularly with the TU/e Executive Board members and with the Supervisory Board to discuss strategical issues regarding also quality of education. The AP educational management department makes use of different information sources to monitor, evaluate and improve education at all levels. The indicators given in the Keuzegids 10, in which all Dutch universities are compared in a ranking, serve as a benchmarking to evaluate the AP study program at national level. Furthermore, the scores of the National Student Survey (NSE), serve as a thermometer to measure the level of satisfaction of the quality of education and study program. In addition, the information provided in the Monitor survey is also a useful tool to measure to what extent the students are satisfied with the AP study program and to what extend students can apply in the working place the knowledge and skills gained in our program. 2.2 Support to faculty staff on improvement of quality of education The AP educational management department considers crucial the support to the faculty staff during the implementation of the improvement plans, as an essential element that contributes to the quality of education. Regarding the implementation of the plans, the faculty staff can get support from the AP educational department with respect to didactics on feedback and assessment, IT-tools and innovation in education (i.e., weblectures/pencasts), redesign of the courses, project assignments, among others. In addition, lecturers are motivated to participate in innovation projects as a mean to get in contact with new tools and teaching methods. Technical support on the classroom response systems, the so-called clickers, and alike tools is provided by central level upon request. Regarding the professionalization of lecturers, there are courses provided by the TU/e at central level on teaching and learning in higher education. These courses can be advised to follow during the discussions with the responsible lecturer(s). With respect to the introduction of the English bachelor program, feedback on English skills can be offered to the faculty staff during the implementation of courses. In addition, the English Language Assessment (ELA) test will be advised to take if needed. 10 The Keuzegids (Selection Guide) is the only independent consumer guide that evaluates and compares the quality of all study programs in the Netherlands. 7 Quality assurance system Applied Physics department /June, 2017
3 Criteria and frequency to monitor the quality of education at course level The AP educational department follows specific criteria to evaluate courses 11 : It is a new course; There is a new teacher responsible for the course; There are (some) items in the course evaluations which score lower than the general standard indicators; The course has low/high pass rates (according to standards); Significant changes have been made to the course as a result of previous evaluation; There is a lecturer involved in the course who follow the UTQ/BKO program; Specific themes need to be followed up and/or evaluated, i.e., PRV; drop-out BC regulations on interim test in second and third year; math learning line, etc.; New departmental or curriculum changes are integrated, i.e., English bachelor study program; Lecturer has introduced innovations (i.e., pencasts, weblectures, etc.); The AP Applied Natural Sciences course must be evaluated every year; Upon request by the responsible and/or co-lecturers. Once first year courses, and subsequently, the second and the third year bachelor courses are evaluated regarding the quality of teaching in English, the AP department will turn back to the traditional system of evaluating courses once every three years, unless one or more criteria mentioned above are met. 11 The criteria followed to evaluate courses is similar to the one followed by other TU/e departments. 8 Quality assurance system Applied Physics department /June, 2017
4 Instruments to monitor and evaluate the quality of education at course and program level The AP education management department makes use of a number of instruments to monitor, evaluate and control the quality of education at different levels and by different actors. In Table 3 a summary of the instruments and the function is summarized. Table 3. Instruments to monitor quality of education Instrument Course level Course evaluation Bachelor teachers and students councils (p-/pp-/mraad) Follow-up students groups (college volggroep) Review meetings faculty staff Program Committee Examination Committee Management reports BC Kroket lunch/monitoring on AP basic course Function and purpose Monitor the quality of the courses, provide feedback to the teaching staff and the students, and inform the internal forums. Feedback on courses running in the quarter at the time of holding the council meetings. Overall purpose is to adjust education on time before course ends. Feedback also on education of courses which have already finished towards improvements for the following academic year. The meetings are organized once per quarter (with the exception of the master council) together with teachers and students and representatives of the AP educational management department. Faculty staff responsible for AP and also for elective courses followed by AP students at other departments are invited to take part of the council meetings. At the beginning of each academic year, the AP education policy and quality assurance advisor holds a meeting with a representative of STOOR. A list of courses to follow up is given. Students groups follow the lectures and instructions and provide the lecturers with feedback on different elements of the course. Follow-up groups are organized by STOOR. The AP education policy and quality assurance advisor holds frequently review meetings with the faculty staff (and/or provide advice and feedback) to analyze course results, monitor progress and advice on improvements. Courses with <60% and >90% pass rate or numerous items scores <3 in the students questionnaires get special attention and test analyses are also discussed with the lecturers. The Program Committee safeguards, among other tasks, the quality of courses and study program and makes recommendations for improvement. The Examination Committee safeguards, among other tasks, the quality of examinations and makes recommendations for improvement. The EC monitors the level of education through internal supervision of the examinations in terms of content, assessment method and level. The AP educational management department elaborates regularly management reports to inform the departmental forums on course (i.e. course performance, students questionnaires) and program performance (i.e., students math level, study delay in BEP and master graduation projects). The different forums elaborate advices to AP educational management department if needed. The BC quality assurance advisor monitors the quality of the BC general courses, among those, the AP Applied Natural Sciences (ANS). The feedback given by the students on this course during the crocket lunches organized by the BC is sent to the AP educational policy and quality assurance advisor. This feedback together with the students questionnaire and by the students council is discussed annually with the AP ANS responsible lecturer. An evaluation report is sent to the BC. The report is discussed both 9 Quality assurance system Applied Physics department /June, 2017
Program level Annual report Semester meetings Bilo Regular meetings with Supervisory Board and Departmental Council BSc Transfer survey (aansluitenquête) BSc end-of-first, second & third year survey (p-/pp curriculum enquête) GS Transfer survey National level Keuzegids/National Student Survey (NSE) TU/e Alumni monitor National Alumni Survey (Nationale Alumni Enquête former WO monitor) in the BSc Program directors and in the Advisory Committee for Bachelor s Education (ACB) meetings. The AP education management department summarizes the most relevant information on the study program in the BSc and MSc annual reports. It provides an overview of past and future actions. Reports are sent to both internal (PC) and external (Deans BC and MSc, CCKO, TU/e Executive Board) stakeholders for accountability. Internal and external bodies may make suggestions on course and program quality of education. TU/e Executive Board, Deans BSc and MSc organize semester meetings with the AP Departmental Board to review, among other issues, the progress of the quality of education and other related educational strategies and plans. The AP Departmental Board organizes regular meetings with the Supervisory Board and with the Departmental Council to review, among other issues, the progress of the quality of education and other related educational strategies and plans. This survey is organized by the Bachelor College (BC) to appraise how first year students have experienced the transfer from high school (VWO) to the university. The focus on this survey is to gain an overview of how organizational and program related issues may influence students perceptions about the study program they have chosen. This survey is organized by the Bachelor College (BC) to appraise how first, second and third year students have experienced the first year of the study program. Questions are related to coaching, study approach, professional skills, etc. among others. This survey is organized by the Bachelor College (BC) to appraise how first year master students have experienced the transfer from bachelor to the master program. The NSE is a large-scale annual survey that invites nearly all higher education students in the Netherlands both bachelor and master to give their views about content of degree program; lecturers; information provision by the program; learning and study facilities; study load; and assessment and grading. The TU/e Alumni monitor is an alumni survey of all TU/e departments which is carried out by the TU/e once every two years (last time in 2015). The National Alumni Survey (formerly: WO Monitor) is a national survey of all newly graduated master's and doctoral students at the (expensed) Dutch universities. It focuses on gathering information on first job and retrospective study. The outcomes provide insight into the connection between the master's program and the labor market. Since 2009, the research has been conducted twice a year among Dutch-funded universities and it is carried out by VSNU. The survey is conducted after October 1 st, and the results are made available to the universities the year after. 10 Quality assurance system Applied Physics department /June, 2017