Quality Assurance: interface between the quality assurance agency and the institution Workshop 3: Good Practices in University Quality Assurance Buenos Aires, 13 th April, 2016 Susanna Karakhanyan, PhD President of ANQA Accreditation Commission, Armenia Vice-President of INQAAHE QA Manager, Abu Dhabi Education Council, the UAE World Bank, Asian Development Bank, UNESCO, EU expert
2 Introduction of Participants Who are you? What are your expectations? What would be of a value added for you? Workshop3: Good Practices in University Quality Assurance
3 Expected outcomes By the end of the workshop the participants will be able to: Discuss the international trends in quality assurance of higher education in general Discuss and critically evaluate quality assurance methodologies and their fit to a particular system Discuss and design self-assessment studies that best meet the needs of the system Workshop3: Good Practices in University Quality Assurance
4 Expected outcomes By the end of the workshop the participants will be able to: Discuss the international trends in quality assurance of higher education in general Discuss and critically evaluate quality assurance methodologies and their fit to a particular system Discuss and design self-assessment studies that best meet the needs of the system Workshop3: Good Practices in University Quality Assurance
Autonomy vs. Accountability 5 AUTONOMY CRUCIAL for SUCCESS: Need to well-balance autonomy with accountability ORGANIZATIONAL FINANCIAL STAFFING ACADEMIC - Select and dismiss executive head - Select criteria for executive head - Decide on GB composition - Decide on academic structures - Create legal entities - Length and type of public funding - Keep surplus - Borrow money - Own buildings - Charge tuition fees - Decide on recruitment - Decide on salaries (senior academic/senior admin) - Decide on dismissal of staff (senior academic/senior admin) - Decide on promotion ACCOUNTABILITY - Decide on overall student numbers - Select students (BA and MA) - Introduce programs - Terminate programs - Choose language of instruction - Select quality assurance mechanisms and providers - Design content of degree programs Autonomy Accountability To whom: - The government - The public at large Tools - Accreditation and authorization - Financial Audits - Key Performance Indicators - External evaluations - Transparent performance - Reporting tools, scorecards 5
F UNCTIONS 6 Functions of national quality assurance framework U N I T S O F A S S E S S M E N T Higher education institution Program Student External quality assurance agency Entry point/ initial assessment - Licensure/authorization - Government authority/ministry - Licensure/authorization - Government authority/ministry - Admission tests - Government decree or authorization - Private provider/ngo Monitoring/ enhancement - Audits/inspections - HEI - HEI - Audits/inspections Accountability/ credibility - Buffer body, - An independent agency, - professional organizations - Buffer body, - An independent agency, - professional organizations - Assessment of learning outcomes - National and international auditing - QA networks - overarching QA standards/good practices Professional Certification - N/A - Professional associations - Government entities - Professional associations - Government entities - N/A Public information - MoE and/or agency - HEIs - MoE and/or agency - HEIs - N/A - National governments - International QA networks 6
Consequ ences Outputs Tools 7 Purposes and types of external reviews Granting rights to function Control Enhancement Accountability Licensure/ Authorization Inspections Monitoring KPIs Audits External evaluation Accreditation License Sanctions in case of failures Reports with recommendations and commendations Yes/No decision Right to function Risk to lose the right to function Conditions to improve Links to funding Right to operate Conditions to improve Workshop3: Good Practices in University Quality Assurance
Freque ncy Outputs Unit of assess ment 8 The main approaches to quality assurance Accreditation Assessment Audit Program Institution Program Institution Internal procedures of an institution to ensure the objectives are achieved Yes/No decision Status Graded judgment (numerical, literal or descriptive) Reports with recommendations and commendations On a regular basis About 5 to 10 years On a regular basis Mainly 5 year-cycle On a regular basis Mainly 5 year-cycle Workshop3: Good Practices in University Quality Assurance
9 Institutional vs. program reviews Institutional review Program review Advantages Disadvantages Advantages Disadvantages - Enables establishment of quality culture in case of robust embedded QA practices - Promotes constructive changes in relation to management and administration - Brings about efficiency, effectiveness, continuous improvement and sustainability in operations - Less costly - Less emphasis on the award of qualifications per program - Such crucial aspects as faculty qualifications, teaching and learning methodology tailored to each program need and the like may be overseen due to a major objective of institutional effectiveness - Looks at the intended and achieved learning outcomes of an individual program - Looks at the fitness of TLA to the learning outcomes - Ensures credibility of a degree/qualification - Not cost-effective - If conducted alone jeopardizes establishment of quality culture at the institutional level Workshop3: Good Practices in University Quality Assurance
Status Reviewe rs QA Body 10 Types of accreditation National International Joint Done by the national agency Done by international EQAAs Done by both national and international bodies - Local peers - Sometimes internationals are invited - Only international peers - International peers - Sometimes locals are invited The panel has equal representation of national and international peers and procedure coordinators Label of the national accreditation body Label of the international accreditation body Labels of both national and international accreditation bodies Workshop3: Good Practices in University Quality Assurance
11 What to consider while opting for a specific type of accreditation NATIONAL LEGAL FRAMEWORK- mandates CRITERIA predetermined and transparent (whose criteria will you follow) DECISION-MAKING whose label will you obtain? PROCEDURE combination of self-assessment and external review (whose procedure) PUBLIC DISCLOSURE -Disclosure THE NATURE of the OF final EVALUATION provision of mere outcomes vs. the full report information vs. in-depth analysis EXPERT PANEL COMPOSITION (Local vs. international experts) VALIDITY PERIOD to ensure Absence a cyclical of evaluation culture approach. International accreditors tend to ask for provision of data and evaluate themselves
12 Approaches to EQA: evolution? Mainly combination of approaches Newly emerging systems Developing systems Usually start with institutional, program or combination of both Developed systems - Move to self-accreditation models with EQAs performing mainly external reviews at institutional levels and delegating program accreditation to the HEIs; - Design new approaches to accommodate the needs (e.g. risk-based, KPI system, value added score) Workshop3: Good Practices in University Quality Assurance
Standards/ guidelines Year of establishment Coverage 13 Overview of major QA networks/associations INQAAHE ENQA CHEA RIACES APQN Worldwide Europe The USA and beyond Ibero- America Asia-Pacific 1991 2000 2003 2003 1996 Guidelines of Good Practice ESGs CHEA Eligibility standards and Principles for HE Internationally Guidelines of Good Practices Chiba Principles Workshop3: Good Practices in University Quality Assurance
Functional Operational 14 International Reference Points OVERARCHING STANDARDS/GUIDELINES INQAAHE GGP (under revision) ESGs (revised in 2015) CHEA guidelines (revised in 2010 and additions are made in 2016) RIACES Chiba Principles Section I: The EQAA: accountability, transparency, and resources Section IV: External activities: collaboration with other agencies and transnational/cross-border education - III. Standards and guidelines for quality assurance agencies Standard B: Demonstrates Accountability Standard D: Employs Appropriate and Fair Procedures in Decision Making Standard E: Demonstrates Ongoing Review of Accreditation Practices Standard F: Possesses Sufficient Resources Section 1: Guidelines for the EQAA: - Mission and purposes, - Organization and resources Section 3: The agency and its environment - Publicity of decisions, - Monitoring of the operation of the EQAA, - Networking and links with other bodies. - Quality Assurance Agencies: key principles guiding the structure of quality assurance agencies and their management if they are to effectively conduct assessments for the accreditation and auditing of institutions and programs. The principles evolve around EQA operations. -Section III: Institutional of higher education and the EQAA: relationship, standards, and internal reviews Section II: EQAA review of institutions: evaluation, decision and appeals I. Standards and guidelines for internal quality assurance of HEIs II. Standards and guidelines for external quality assurance of HEIs Standard A: Advances Academic Quality Standard C: Encourages, Where Appropriate, Self- Scrutiny and Planning for Change and for Needed Improvement Section 2: The agency s review processes: - Relationship with HEI, - External review procedures (Quality criteria, procedures, external review teams); - Documentation, decision making process. - Institutional Quality Assurance: key principles guiding institutions in assuring their own quality. The principles evolve around two major domains: 1. Internal quality assurance of HEs, and 2. external quality assurance of HEs. Workshop3: Good Practices in University Quality Assurance
15 Expected outcomes By the end of the workshop the participants will be able to: Discuss the international trends in quality assurance of higher education in general Discuss and critically evaluate quality assurance methodologies and their fit to a particular system Discuss and design self-assessment studies that best meet the needs of the system Workshop3: Good Practices in University Quality Assurance
16 Popular methodologies Methodologies Peer review Risk based assessment KPIs Benchmarking Diversity of metrics: Value added score Gainful employment Combination of several tools linked to the system needs The best practices are based on the in-depth understanding of the system needs Workshop3: Good Practices in University Quality Assurance
17 Peer-review A panel of external reviewers (usually peers in the field) A collective responsibility based on mutual respect and a sense of common purpose Staff with knowledge of quality assurance in their own institutions Objective assessment and informed judgments Training and briefing to ensure consistency in evaluations Working within an agreed framework of reference Advantages - The methodology provides for both detection of areas for further improvement and good practice. - Qualitative analysis based and supported by qualitative and quantitative indicators - Supports detection of problems with the processes, rather than only outcomes - Helps to identify problems before they arise by using robust, cyclical checks on institutional policies, rather than waits for a problem to arise and then act. Disadvantages - After a couple of cycles loses its initial effects - Most depends on the capacity of the reviewers and the procedure coordinators - Subjective depending on the capacity of the reviewers Workshop3: Good Practices in University Quality Assurance
18 Risk based assessment A quality regime established and run by the government on an annual basis to monitor and detect risks Focuses regulatory efforts on the risks detected Offers a differentiated approach to the variety of providers, including the scope, duration, validity of external evaluation Advantages - Less costly - Less bureaucracy, less burden on advanced HEIs - Is tailored to the needs of individual providers Disadvantages - Mainly quantitative, which makes it difficult to measure the quality of learning provisions - Problems are detected and resolved only after a period of time of a downward trend, rather than on ongoing basis as in case of more frequent scrutiny; - There is a risk of categorization of HEIs - Less opportunity to identify the good practice and disseminate throughout the system, including innovations in teaching and learning, student assessment and the like Workshop3: Good Practices in University Quality Assurance
19 Key performance indicators Usually established by the governments, buffer bodies or HEIs as a tool for informing the general public or government on the status of the system; monitoring the progress of a policy, strategy, or plan that has been implemented (or one of its components); managing the higher education system or an institution as a whole. Advantages - Promotes a culture of objective data collection and analysis - Promotes informative decision-taking - Regular publications ensure transparency with the most recent data Workshop3: Good Practices in University Quality Assurance Disadvantages - Fails to capture qualitative aspects of teaching and learning; - Problems are detected and resolved only after a period of time of a downward trend, rather than on ongoing basis as in case of more frequent scrutiny - Less opportunity to identify the good practice and disseminate throughout the system, including innovations in teaching and learning, student assessment and the like
20 Some examples for KPIs: Gainful Employment Gainful employment a condition whereby an employee has a consistent work situation and is consistently paid by an employer so that they may make ends meet. Psychologically speaking GE is pursuing a career path that brings satisfaction and purpose to individuals lives and the work they accomplish. Educationally speaking: GE has become a key topic in education reform and is now used as a Quality Indicator. Workshop3: Good Practices in University Quality Assurance
21 A Recent Perspective on GE (from the US Government Perspective) According to the US Department of Education, the new Gainful Employment Regulations requires schools: to provide students with an education that will lead to employment in a recognized occupation whereby they may earn a wage or salary adequate enough for them to pay their loans back after they graduate from college. Workshop3: Good Practices in University Quality Assurance
22 In the US as of July 2015 Under the new regulations a program would be considered to lead to gainful employment if: - The estimated annual loan payment of a typical graduate does not exceed 20 percent of his or her discretionary income what is left after basic necessities like food and housing have been paid for or 8 percent of his or her total earnings. Programs that exceed these levels : - Would be at risk of losing their ability to participate in taxpayerfunded federal student aid programs. Workshop3: Good Practices in University Quality Assurance
23 Expected outcomes By the end of the workshop the participants will be able to: Discuss the international trends in quality assurance of higher education in general Discuss and critically evaluate quality assurance methodologies and their fit to a particular system Discuss and design self-assessment studies that best meet the needs of the system Workshop3: Good Practices in University Quality Assurance
24 Factors and principles influencing the quality of interaction and outcomes Factors Principles National priorities/problems to be solved The design of the QA framework The criteria and procedures The review model The external reviewers The EQA leadership The consequences tied to the QA mechanisms Inclusiveness Credibility Transparency Consistency Independence Workshop3: Good Practices in University Quality Assurance
25 - Agency coordinators - External reviewers - Procedures Quality Assurance: the interface between the quality assurance agency and the institution Self evaluation - Standards, criteria, guidelines Follow-up Internal preparation Action Plans External review - Agency coordinators - External reviewers - Procedures Outcomes and reflections - Commendations and Recommendations - Conditions for enhancement - Financial or regulatory Workshop3: Good Practices in University Quality Assurance implications
26 What is self-assessment? Definition Purpose Self-assessment is an ongoing activity conducted by the institution or an academic programme to assess whether the educational services provided by the university /academic programmes meet their educational objectives and outcomes. To improve the intentions (purposes and goals), content, policies, procedures, services, organisational and intellectual environment and performance of the program or institution under study. If done with commitment and a wider excellence vision it will lead to the establishment To foster of commitment an organizational by enacting the recommended improvements through participation in the study excellence in To provisions. enhance the capacity of the program or institution in question for continued self-assessment. learning culture leading to continuous enhancement and To yield the basis for informed decision-making (planning) about the future of the unit under study. To yield written materials that can be used as the basis of external peer review and review by others within the larger institution or system.
27 Who should be involved in the SA process? Self-assessment document Who can do it effectively? A guide to the institution/programme area A self-critical assessment not just a description Identifying issues for consideration, with proposed actions A shared document representing a collective view Cross-referenced to sources of evidence Dedicated team: Faculty members and staff Students, alumni External stakeholders VS. Hired consultants But, the desired impact can only be achieved with an inclusive approach of internal and external stakeholders
28 Major steps in self-assessment Purpose identification Outcomes identification Measurements and evaluation design Data collection Analysis and evaluation Decision-making on actions to be taken Why do we need to engage in selfassessment? What results do we need to achieve with the selfassessment? What methodology, tools and mechanisms for measuremen t should we use? What type of data do we need to obtain? What are the methods we are going to apply to analyse the data? How are we going to use the results of the analysis in the decisionmaking regarding new strategies and developments?
29 How to put together a valid self-assessment report: questions to be answered Which indicators do I need to control my process? Which indicators are out of my influence? What indicators do we have already? Which indicators do I need for doing the work? What is the costbenefit ratio of collecting indicator data? Which indicators do we need for improvement and enhancement?
30 Major sources of data Performance outcomes: Stakeholder feedback: Performance outcomes Feedback (samples) mechanisms, and other evaluation groups, which evidences (thesis provide papers, for exam qualitative questions Statistical information; and data: respective answers, e.g. samples questionnaires, of publications, which are a powerful observations of classes element and the if devised like) the right way; discussion-oriented team work quality circles, peer-reviewers (external and internal); focus groups, individual interviews and the like Available at the university units and/or statistical office, that is key performance indicators (KPIs) (data Benchmarking: like student success rate, drop-out rate, in-coming out-going students, number of Provides teaching a staff, firm background budget for justification of allocations and the like) current approaches and aspirations for the future
31 Mapping a process Feedback mechanisms Statistics Benefits Process Sub-process Sub-process Policy & procedure Policy & procedure Performance evidence Benchmarking Feedback mechanisms Statistics By mapping each major process in HEI operations and functions a clearer picture with regards to the state of affairs will surface Performance evidence Benchmarking
32 Sample indicators for quality assurance of teaching and learning Input indicators Process indicators Output indicators Student satisfaction (to be refined) Teacher satisfaction (to be refined) Awareness of objectives Information collected by the university units (database information) Student satisfaction (to be refined) Teacher satisfaction (to be refined) Learning outcomes Learning gain Assessment of students progress rate Student satisfaction (to be refined) Teacher satisfaction (to be refined) Employers satisfaction (to be refined) Alumni satisfactions Employment rate Right after graduation 1 year after graduation five year after graduation went-on with research in another profession rather than the degree
33 Sample indicators for quality assurance of teaching and learning Input indicators Process indicators Output indicators Student services Benchmarks with other universities Drop out rate In-coming and out-going students Information collected by the university units (database information) Student services Benchmarks with other universities Learning outcomes Learning gain Assessment of students progress rate Information collected by the university units (database information) Student services Benchmarks with other universities
34 Quality assessment: some methods Methods of Analysis Process flow-chart Graphs Pareto analysis Fish-bone diagram Scatter diagram Check sheets Control charts Brainstorming Benchmarking SWOT analysis 140 120 100 80 60 40 20 0 2008 2009 2010 2011 Programme 32 Programme 2 Programme 1 Syllabi Learning Teaching outcomes and learning methods Assessment methods Organization and management Advising services Learning resources Research Curriculum Research Learning culture outcomes Budget Publicatio Workload Planning ns Librar ies Labs External examiners Assessment methods Student assessment Learning methods Teaching methods Teaching and learning Internal and external review Admissio n Teacher qualifications Quality Academic program Monitoring students progress
35 Sample format for self-assessment - One page The history text and should current be developed profile of the per criterion/standard institution; and include: - An overview - of In-depth the management analysis of quantitative and organizational and qualitative data; structure, including - Factual an information organization not chart available (or in other materials (such as equivalent); annual reports), Appendices cross- referenced should to, indicate: and within, the main - A brief overview document; of the institution s - how this quality information assurance is used in quality control and arrangements - Effectiveness and systems, of a assurance particular processes approach; within the institution, giving examples - The scope of - the How audit, this information where possible; is used in quality control and - The fitness of QA assurance arrangements - processes examples for within the of the context the effectiveness institution, of giving in the examples operation of quality the institution's where charter, possible. profile/objectives assurance systems and and instances of particularly good special characteristics. - Examples of practice; the effectiveness in the operation of quality assurance - systems The referenced and instances documents of particularly (e.g. strategic good plan, yearly and practice other plans ); - SWOT analysis. - Any other attachments/evidences to justify the case. Introduction The main document Annexures
36 Sample text The analysis of alumni survey data corresponding to learning outcomes are presented in Fig. 13 and Fig. 14. It can be seen from the results of the alumni survey that most of the learning outcomes are being achieved at above satisfactory level. However, the following outcomes are lower than the others: - Identify and evaluate organizational requirements with the current and emerging technologies. (3.75/5.00) - Communicate effectively, both in writing and in speaking. (3.76/5.00) - Follow the latest developments within the field of IT. (3.72/5.00) These results show that additional attention should be paid for these particular outcomes. On the other hand, the results given in Fig. 14 for alumni who graduated within the last year, exhibit remarkable increases in the averages for all outcomes showing the progressive improvement.
37 The most important message Three things matter most: Leadership Direct implementers and Stakeholders If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea. - Antoine de Saint Exupéry - Ivan Aivazovsky - 1867 Workshop3: Good Practices in University Quality Assurance
38 Break up session (1 hour) Break up into groups to discuss: Which methodologies best fit your system? What would you change/combine to make the methodology better meet the needs of the system? What are the motivators for establishment of a quality culture at your institution? How could this be done? What support you might need? Workshop3: Good Practices in University Quality Assurance
39 Reflections What do you now know that you didn t know before this workshop? What would you do differently now? What is the way forward? Workshop3: Good Practices in University Quality Assurance
Thank You 40 s.karakhanyan@gmail.com s.karakhanyan@anqa.am Workshop3: Good Practices in University Quality Assurance