Chapter 5: Academic Program Quality Review Contents Structures & Responsibilities 5.1 Programme Quality Review Framework 5.2 Programme Learning Outcome Assessment 5.5 Annual Academic Programme Audit 5.13 Cyclical Programme Review 5.17 Policy and Procedures 5.21 Structures & Responsibilities Office of the Deputy Vice Chancellor Academic Affairs Program and Curricula, Academic Affairs The is responsible for supporting Academic Faculties in the continuous quality improvement of the HCT s educational provision, the development of the academic quality program review provision, and the assurance of its implementation. The is headed by the Assistant Deputy Vice-Chancellor Program and Curricula and contains a specialist section for Program Review and Improvement. Academic Faculties Faculty Academic s (FAC) The FAC is responsible for implementing the policy and procedure related to program review. Industry Advisory (IAC) Industry Advisory s (IAC's) provide input from local industry professionals and other external stakeholder into the programme and its courses from the perspective of industry professionals and external community stakeholders. Academic Council The Academic Council receives and evaluates recommendations from the Curriculum Quality Assurance and endorses them for approval by the Executive or Board of Trustees. System-wide Curriculum Quality Assurance (CQA) The oversees and supports the implementation of the program quality review provision, and reports annually to Academic Council on the effectiveness of program quality review processes and activities. The approves the Cyclical Program Review calendar. Board of Trustees The Board approves the teaching-out of an academic program across the HCT. Institutional Research Unit, Organizational Excellence (IR) IR provides input in terms of institutional research data and analysis, and monitors the effectiveness of the academic program quality review provision. IR also publishes an annual Institutional Effectiveness (IE) Report which contains Key Performance Indicators (KPIs) for Academic Faculties and campuses. The Report provides trend analysis of institutional effectiveness indicators over the last three years, responses from Academic Faculties and campuses on the past trends, targets and action plans for improvement. 5.1
Programme Quality Review Framework Purpose The HCT s academic Programme Review Policy is designed to ensure that programmes are: Aligned to the Mission of the HCT Regularly enhanced to meet the changing needs of the nation and its economy Continuously improved for effective learning Upgraded to align with new and emerging technologies Effectively resourced in terms of instructors, teaching and learning resources, computer and information technology, physical facilities, and budgeting Requirement Academic Programs Process Frequency Focus Timeline HCT 2.0 Bachelor s, Higher Diploma, Diploma Program Learning Outcome Assessment Each PLO at least once every 3 years Minimum 30% of PLOs per year Teaching & Learning August HCT 2.0 Bachelor s, Higher Diploma, Diploma Annual Program Audit Yearly review of key indicators Program Health Check October Bachelor: every 5 years HCT 2.0 Bachelor s, Higher Diploma, Diploma Cyclical Program Review Higher Diploma: every 4 years Program Continuance December Diploma: every 3 years HCT 2.0 Bachelor s, Higher Diploma, Diploma External Accreditation of Academic Programs Periodically (depending on external accreditation schedule) International Standards - 5.2
Overview of the HCT Programme Review Process Year 1 =>30% PLOs Assessed Annual Programme Audit Report Year 2 =>60% PLOs Assessed Annual Programme Audit Report Year 3 100% PLOs Assessed Annual Programme Audit Report Year 4 =>30% PLOs Assessed Annual Programme Report Year 5 =>60% PLOs Assessed Annual Programme Report Cyclical Review Focus Student Learning Programme Modification Programme Continuance Responsibilities: Programme Quality Review Framework Program Review Component Program and Curricula Curriculum Quality Assurance Academic Council Program Quality Review Framework Implements the framework Reports on the implementation of the Programme Quality Review Framework to the CQA Recommends measures for improvement of the Programme Quality Reports on the effectiveness of the Programme Quality Review Framework including Program Learning Outcome Assessment, Annual Program Audits, and Cyclical Program Review Recommends measures for improvement of the Reviews and monitors the effectiveness of Programme Quality Review Framework Receives feedback from Academic Faculties regarding the effectiveness of the Programme Quality Review Framework Receives reports on the implementation and effectiveness of the Programme Quality Review Framework from CQA Approves measures for improvement of the Programme Quality Review Framework 5.3
Program Review Component Program and Curricula Curriculum Quality Assurance Academic Council Review Framework to the CQA. Programme Quality Review Framework to the CQA. Recommends measures for improvement of the Programme Quality Review Framework to the Academic Council Timeline What When Purpose Programme Learning Outcome Assessment At least once every 3 years Continuous quality improvement Program Audit Report Annually Continuous quality improvement - evaluate program s effectiveness in meeting its mission, and the needs of students and society Cyclical Program Review Every 5 years Continuance at the HCT or at a particular campus External Review of Academic Programs Periodically (depending on external accreditation schedule) National and international accreditation Program Review Site Resources and Reports related to program learning outcome assessment, annual program audit and cyclical program review are published on-line. For further details visit the: Program Review Site 5.4
Programme Learning Outcome Assessment Purpose The purpose of PLO Assessment is to monitor the effectiveness of the curriculum and to guide quality improvements to programs. There are two types of PLO assessment: Summative PLO Assessment The main purpose of institutional PLO Assessment is to gather evidence of the cumulative effect of the program curriculum on student learning. Assessments selected from courses close to graduation Courses identified in the Curriculum Map as where students would be able to demonstrate their mastery of the target PLO. Formative PLO Assessment The focus is to provide faculty and students with valuable feedback on progress towards achieving PLOs. The administers throughout the program length employing course assessments aligned to the specific PLO. PLO Cycle Evidence of Student Achievement There are two types of evidence collected: Direct evidence of student achievement of programme learning outcome using embedded course assessments. Academic Faculties may also employ alternatives to embedded assessment such as specially designed assessment tasks, external examinations etc. Indirect evidence of student achievement of programme learning outcomes is collected by means of surveys. The survey results and their analysis form part of the Annual Program Audit Report (see below). Surveys used include the Student Exit Survey, Graduate Employment Survey, Graduate Employer Survey. Requirement For each academic program: Each programme learning outcome is to be assessed at least once over a three year period. A minimum of 30% of programme learning outcomes must be assessed each academic year. Each identifies assessments to provide evidence of student achievement of each Programme Learning Outcome (PLO). Faculties then interpret the results and provide an action plan to address any shortcomings. Typically, the assessments used to provide evidence are assessments from towards graduation e.g. courses in year 3 or year 4 of the programme. The designated assessments should be common assessments i.e. the assessment instrument (oral defence, portfolio, project, report, test etc.), marking scheme or rubric, grading and standards of achievement should be the same for all students taking the assessment across the system. Academic Faculties must ensure that the following are kept for example in the relevant electronic Course File (ecaf): Copies of the assessment instrument Marking scheme/rubric Details of moderation process Samples of student work reflecting the range of performance - wherever possible Quantitative analysis of results 5.5
Process Long Term (3-Year) PLO Assessment Plan Modifications to Plan Annual PLO Assessment Plan Evaluate impact of action plan Assess students PLO Assessment Process Implement Actions for improving student learning Report Results Action Plan for improving student learning Program & Curricula Review Report & Action Plan 5.6
Long Term (3-Year) Assessment Plan Continuous Quality Improvement Cycle ASSESS EVALUATE Action Plans for CQI CHANGE Implement Action Plans ASSESS Evaluate Change Year 1 Year 2 Year 3 Year 4 Sample PLO Schedule (ABET Inc.) Curriculum Maps Course learning outcomes are mapped to programme learning outcomes to show how each course contributes to student achievement of the outcomes. The mappings monitor that each programme learning outcome is adequately covered through the curriculum and that all courses are aligned to the outcomes. Mapping reports are available from the. Sample curriculum map 5.7
Level 5 Level 6 Level 7 Course PLO1 PLO2 PLO3 HSW 1003 Introduced HSW 1023 Introduced Introduced HSW 1033 Introduced Introduced HSW 1223 Introduced Reinforced HSW 1233 Introduced Reinforced HSW 1313 Introduced Reinforced Reinforced HSW 2013 Introduced Reinforced Reinforced HSW 2033 Introduced Reinforced HSW 2133 Reinforced Reinforced HSC 2203 Reinforced HSW 2143 Reinforced Reinforced HSW 2323 Reinforced Reinforced Reinforced HSW 2333 Reinforced Reinforced Reinforced HSW 3013 Reinforced Reinforced Reinforced HSW 3103 Reinforced Reinforced Reinforced HSW 3023 Reinforced Reinforced HSW 3033 Reinforced Reinforced HSW 3223 Reinforced Reinforced Reinforced HSW 3943-WP Reinforced Reinforced Reinforced HSC 4003 Reinforced HSW 4223 Reinforced Reinforced Assessed/Mastered HSW 4233 Assessed/Mastered Assessed/Mastered Assessed/Mastered HSW 4033 Reinforced HSW 4243 Reinforced HSW 4928 Assessed/Mastered Assessed/Mastered Assessed/Mastered HSC 4006 Assessed/Mastered HSW 4303 Assessed/Mastered Assessed/Mastered Assessed/Mastered (Bachelor of Social Work) Method Each develops a long term plan for each program to ensure that evidence is collected of student achievement for each programme learning outcome at least once every 3-years by a: Direct assessment in a required course such as a final assessment, capstone project, external assessment etc. Common assessment instrument (oral defense, portfolio, project, report, test etc.), marking scheme/rubric, grading system, moderation process, and standards of achievement. Academic Faculties select the most valid assessments to be used as evidence of student achievement of programme learning outcomes from year 3 or year 4 BAS courses, and from year 2 Diploma courses. Best practice indicates that Faculties should identify a number of different assessment tasks to provide for a wider sampling of student learning allowing students to demonstrate a wider range of knowledge, skills and competencies than can be observed from reliance on a single form of assessment such as a written examination. 5.8
The designated assessment must provide sufficient depth and scope to produce valid evidence of student achievement of the targeted programme learning outcome. In cases where no single assessment is sufficient, a number of assessment may be designated. Each designated assessment becomes a common assessment i.e. the assessment instrument (oral defence, portfolio, project, report, test etc.), marking scheme or rubric, grading and standards of achievement are the same for all students taking the assessment across the system. Illustration: 3-Year PLO Assessment Plan PLO Programme Learning Outcome 2017-18 2018-19 2019-20 1 An ability to apply knowledge of computing and mathematics appropriate to the discipline CIS 3003: FWA 2 An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution 3 An ability to analyse a problem, and identify and define the computing requirements appropriate to its solution. CIS 4906: Capstone project CIS 4906: Capstone project 4 An ability to function effectively on teams to accomplish a common goal CIS 4103: CSA 5 An understanding of professional, ethical, legal, security and social issues and responsibilities CIS 3103: CSA etc Annual Programme Learning Outcome Assessment In each academic year, it is expected that for each program, evidence of student achievement will be collected for a minimum of 30% of each of programme s learning outcomes. Process Step Process When By Action Plan Develop assessment plans for each PLO Designate assessments for each PLO Start of academic year Faculty Academic Submit to Executive Dean to review and recommend for approval by Program & Curricula Do Implement assessment plan consistently across classes and delivery sites Throughout academic year (Course Team) Administer assessments Collect data from direct assessment Throughout academic year (Course Team) Mark and record results 5.9
Check Analyze data Identify shortcomings and/or anomalies Each Fall and Spring semester Input to Annual Program Audit and Program & Curricula Develop action plans to address shortcomings and/or anomalies End of each semester (Course Team) Minor modifications: submit to Executive Dean for approval Major modifications: submit to Executive Dean to review and recommend for approval by Academic Council Input into Annual Program Audit Act Implement action plans e.g. modify curriculum, assessment, resource allocation (instructors, learning resources, physical resources, budget) Following semester (Course Team) Update: curriculum management system, programme and course documentation Monitor effectiveness of actions. Report effectiveness of QA process Throughout academic year and Program & Curricula Input to Annual Program Audit Annual Programme Learning Outcome Assessment Plan and Report The Plan describes how the Long Term Plan will be implemented in the academic year and the standard to be used for evaluating student achievement. When completed, the Report provides the results of the assessment used as evidence, the analysis of the results, and plans to deal with any shortfalls in meeting expectations (e.g. issues related to the assessment task used etc.) and to improve student learning (e.g. modifications to teaching & learning strategies, materials etc.). Annual Program Learning Outcome Assessment Report The template provides: 1) the assessment plan for the year 2) when completed forms the annual assessment report The template is completed to report assessment results for the system as a whole and then to report the results campus by campus. Rows can be added to report multiple assessments for a single PLO including additional direct assessment or indirect assessments (e.g. surveys). 5.10
System-wide Summary PLO No. PLO wording Course code Assessment Task/Instrument Target Location All Campuses No. of Students Result Analysis Areas for Action Breakdown by Campus - add a new row(s) for each campus as required. PLO No. PLO wording Course code Assessment Task/Instrument Target Campus No. of Students Result Analysis Areas for Action Notes Target: the default (80% of students achieving standard) may be changed to reflect the academic judgment of the level of challenge of the assessment task. Analysis: Focus should be on identifying areas for improving student learning e.g. curriculum, delivery, resources, student progression etc. Areas for Action: Summarise key areas for improvement. Use the separate Action Plan Template to provide detail & timelines. Important: a 'Timeline' is required for each action. Actions to improve the assessment itself or target set affect the quality of the process and do not improve student learning. They should be noted here but not be included in the detailed Action Plan. Areas for Action Actions for improvement are the key outcomes of the assessment process and should be completed for the System and separately by each campus. The purpose of these actions is to improve student learning and provide evidence to stakeholders and accreditation agencies of the use of assessment results to improve the quality of the academic program and to enhance student learning. Each area for action should have its own Action Plan which provides further details and timelines of the actions to be taken. Action Plan Template Objective Priority Action to be taken Success indicator(s) Owner Target date Status Evaluation 5.11
Responsibilities: Programme Learning Outcome Assessment Program Learning Outcome Assessment (PLOA) Program & Curricula Initiates the PLO Assessment process Provides PLOA templates for plans and reports Reviews Faculty PLOA plans Reviews PLOA Reports from Academic Faculties Publishes PLOA Plans and Reports Approves any update/change to published plans The Faculty Executive Dean with the support of the Faculty Academic and faculty members: Submits a 3-year PLOA plan for each program for review by Programme and Curricula Ensures that 100% of PLOs are assessed over a 3 year period with at least 30% of all PLOs assessed in each academic year Produces an Annual PLOA plan for each Diploma, Higher Diploma and Bachelor program Implements the annual assessment plan consistently across classes and delivery sites and collects data Analyzes data and develops action plans Reports results and analysis and action plans to Programme and Curricula Ensures that an electronic file containing copies of each PLO assessment and samples of student work reflecting range of performance is maintained Ensures that action plans to improve program quality are implemented and evaluated Timeline Action Responsibility When Annual PLO Assessment Plan Draft By end of March Plan Endorsement Program & Curriculum By end of April PLO Assessment Administration Fall & Spring Semesters Annual PLO Assessment Report & Action Plan completed Review of Program Reports & Action Plans Program & Curriculum By end of August By end of September 5.12
Annual Academic Programme Audit Purpose The purpose of the Annual Academic Program Audit is to evaluate key indicators to identify any issues and take action to improve program quality. Requirement Each is required to complete an annual internal review for each programme and major under its responsibility. The report is a critical evaluation of: Need for the programme by business and society Demand for the programme by students Curriculum Student learning Faculty Resourcing Continuous Quality Improvement Processes Process Annual PLO Assessments Annual Program Audit Report Review program quality indicators Evaluate actions for quality improvement Implement Actions for quality improvement Academic Council Approve Actions for quality improvement Annual Program Audit Report Process Curriculum Quality Assurance Endorse Report & Action Plan Recommend quality improvement measures Action Plan to improve program quality Program & Curricula Review Report & Action PLan 5.13
Responsibilities: Annual Program Audit Program Review Component Programs and Curricula Academic Faculty CQA Academic Council Annual Program Audit Initiates the Annual Program Audit process Ensures the implementation of the Annual Program Audit Report process Provides templates for Annual Program Audit Reports Endorses the nominated Audit Team for each programme Reviews the Annual Program Audit Report for each program The Faculty Executive Dean with the support of the Faculty Academic and faculty members: Identify an Audit Team for each programme Conduct Annual Program Audit critically evaluating the need, demand, resourcing and quality for each programme Complete Annual Program Audit Report including Action Plan for improvement for each programme Recommends academic programs to undergo immediate cyclical program review to the Academic Council Recommends measures for improving programme quality to Academic Council Receives reports from the Curriculum Quality Assurance Approves academic programs to undergo immediate cyclical program review to the Academic Council Endorses recommendations for improving programme quality Publishes all Annual Program Audit Reports Recommends academic programs for further review to the Programme Quality Review Provides an Annual Academic Affairs Program Audit Report to the Programme Quality Review Submits an Annual Faculty Program Audit Report reviewing & summarizing programme reports to Program and Curricula Implements Action Plan and recommendations from Academic Council for improving programme quality 5.14
Timeline Action Responsibility By end of Annual Programme Audit Report completed for each active programme and major Faculty Executive Dean October 2017 Annual Faculty Executive Summary completed Faculty Executive Dean November 2017 Review Annual Academic Affairs Report completed Recommendations for improving programs and review process Manager Program Review and Improvement Executive Dean Program and Curricula Curriculum Quality Assurance December 2017 January 2018 February 2018 Endorsement of CQA recommendations Academic Council March 2018 Evidence Key Focus Area Student Learning Main Evidence Program learning outcome assessment reports Satisfaction surveys: Exiting Students, Graduates (alumni), Graduate Employers Curriculum: currency & relevance Satisfaction surveys: Graduates (Alumni), Graduate Employers; Faculty Evaluation (Program Effectiveness Survey) Industry Advisory / Industry Advisory Board Feedback Cohort Analysis Demand/Need for the program Institutional Research Data: Student enrolment, attrition, at-risk students, CGPA, graduation rate Employment Rates; Satisfaction surveys: Exiting Students, Graduates (alumni), Graduate Employers; Industry Advisory / Industry Advisory Board Feedback Feedback from learners Resources: faculty, teaching & learning, facilities Satisfaction surveys: Exiting Students, Graduates (alumni Faculty Complement analysis, Workload data Faculty Evaluation (Program Effectiveness Survey) Satisfaction Surveys: Student Services, Staff Services Sources of Evidence Programme Learning Outcome (PLO) Assessment Reports PLO Assessment Reports are available from the refer above for further detail. The requirement is: Each programme learning outcome is assessed at least once over a three year period A minimum of 30% of programme learning outcomes are assessed each academic year. 5.15
Assessment Reports Reports of student performance on course assessments are published after each Semester either on the Portal egradebook or the Academic Programme Review Dashboard of the IEMS. The following reports are available: egradebook Reports The following are available through the HCT Intranet Portal and report the median, mean, and distribution of grades for coursework and final assessments: CSA (Common Specification Assessment) Analysis Report by CSA Analysis Report by campus CSA Analysis Report by course CSA Analysis Report by CRN Course Learning Outcome Report (under development) Dashboard Reports The following are available through the Academic Programme Review Dashboard of the IEMS and report grade distributions and grade point averages by course and programme: Grade Distribution Report by Grade Distribution Report by Course Average Course Grade Point Average Cumulative Grade Point Average by Programme Cohort Analysis The following data are available through the Academic Programme Review Dashboard of the IEMS: Enrolment Attrition rate Average class absences Students on probation Graduation rate Employment rate Surveys The following surveys are available through the Academic Programme Review Dashboard of the IEMS: Internal Student Satisfaction with Programme (Every Term) Campus Programme Chair Survey (Every Year) External Graduate Satisfaction Survey (Every Term) Graduate Employer Survey (Every Year) External Stakeholder Input The following are available from the : Industry Advisory feedback Accreditation reports 5.16
Cyclical Programme Review Purpose The purpose of the Cyclical Program Review is to decide on whether to continue offering the program or to teach-out the program at the HCT or at particular campuses. Requirement Each active programme and major is required to undergo a periodic review notwithstanding any national or international accreditation. However, Academic Faculties are recommended to synchronise the HCT s Cyclical Program Review with external program accreditation reviews wherever feasible. The Cyclical Program Review is a critical evaluation of the academic quality and demand for the program over the previous years and is based on evidence and trend analysis provided by the Annual Program Audit Reports produced in that period. Programmes are periodically reviewed according to length of credential: Two Year Diplomas: every 3 years Three Year Higher Diplomas: every 4 years Four Year Bachelors: every 5 years Two Year Masters: every 3 years 5.17
Process Annual PLO Assessments Annual Program Audit Reports Self-Study Bachelor s: every 5 years Higher Diploma: every 4 years Diploma: every 3 years Decision Continuance - Academic Council Discontinue at campus(es) Executive Close program - Board of Trustees External Review Team Report Cyclical Program Review Cycle Academic Council Recommendation Program & Curricula Review Curriculum Quality Assurance Review Self-study The responsible completes a Self-Study including analyses of its: Strengths, weaknesses, opportunities, and threats (i.e. SWOT analysis); Need by industry and society; Demand by students; Academic quality e.g. are students achieving the programme learning outcomes; are academic standards consistently maintained, are faculty delivering the program appropriate qualified and experienced etc; Budget efficiency. For further information refer to the Program Review Site 5.18
External Review Team Report The External Review Team (ERT) is appointed by the Executive Dean Programme & Curriculum in consultation with the relevant Executive Dean. The ERT: Verifies and evaluates the Self-study. Makes site visits - depending on the nature of the program and the evidence presented in the selfstudy, the ERT, after consultation with the ADVC Programme & Curriculum, may make site visits to all or a sample of campuses and delivery sites to evaluate facilities and resources. Interviews program managers, faculty, current students, alumni, and external stakeholders (e.g. employers), as feasible. Makes a recommendation for continuance of the program. 5.19
Responsibilities: Cyclical Program Review (CPR) Programs and Curricula Academic Faculty External Review Team CQA Academic Council Executive Board of Trustees Initiates the CPR process Ensures the implementation of the CPR process Provides templates for CPR Approves the review team responsible for writing the Program Self-Study Report Approves the External Review Team responsible for reviewing the Program Self-Study Report and conducting campus visits Reviews the CPR Report for each academic program Recommends the internal team to write Self-Study Team for each programme under review Recommends the External Review Team to evaluate the program Completes Program Self- Study Report including critically evaluation of previous Annual Program Audit Reports for each programme under review Makes a recommendation on the continuation of the program Implements recommendations from Academic Council for improving programme quality Critically evaluates the Program Self-Study Report Conducts campus visits evaluating teaching and learning resources, facilities, and interviewing stakeholders including students and faculty Makes a recommendation on the continuation of the program Approves the timetable for the CPR of each academic program Receives and reviews CPR Reports Makes recommendations for program continuance to Academic Council Makes recommendations for improving programme quality to Academic Council Approves continuance of an academic program Recommends discontinuation of an academic program at one or more campuses or across the HCT Endorses recommendations for improving programme quality Approves discontinuation of an academic program at one or more campuses Approves discontinuation of an academic program across the HCT 5.20
Timeline Action Responsibility By end of Calendar for program cyclical review Programmes nominated in addition to the calendar cyclical review CQA CQA April May Programme Self Study Completed Faculty Executive Dean October Programme Quality Review Completed External Review Team November Recommendation Programme & Curriculum December Endorses recommendations Academic Council January Continuance: Executive Programme Review Decisions Change and areas of improvement made to programme and courses Discontinuance at a campus: Executive Discontinuance across HCT: Board of Trustees Faculty Executive Dean February As appropriate Policy and Procedures LP225 Course & Program Review 5.21