Protocols: Course Quality Assurance Protocols to accompany the policy for (Manual of Policy and Procedures C/4.6) January-June 2013
Topics 1. Overview... 3 2. Course Quality Cycle... 3 a. Individual Unit Reports... 5 b. Individual Course Reports... 5 c. Strategic Faculty Courses Update and Performance and risk checkpoints... 9 d. Consolidated Courses Performance Report... 10 3. Additional components of quality assurance... 12 a. Faculty corporate reviews... 12 b. External accreditation... 12 4. Roles and responsibilities... 12 Attachment 1: Model for identifying high performing and underperforming courses... 15 Attachment 2: Suggested format for ICR Summary Report... 16 Versions Date Updates Version 1 November 2009 Original version to accompany revised policy Version 2 June 2010 Amended web links Version 3 Jan 2011 Updated for 2011, including terminology and methodology for higher risk courses, and changes to timelines Version 4 March 2012 Updated for 2012 with changes to timelines and alignment to updated policy. Version 5 April 2012 Updated for 2013 to align with updates to Section 4.6.5 of policy. 2
1. Overview QUT is committed to ensuring that its courses are relevant, current and provide students with a high quality learning experience. A robust quality assurance system is integral to maintenance of the quality of the University s courses. Course quality assurance at QUT is a continuous process, incorporating regular monitoring, review, benchmarking and improvement of the University s courses. QUT has implemented a Course Quality Cycle that incorporates annual review of all courses to provide a framework for monitoring the quality of the learning and teaching experience. The Course Quality Cycle is designed to: facilitate the provision of high quality curriculum and pedagogy that delivers recognised academic standards; enhance strategic outcomes by providing timely identification of high performing and underperforming courses; facilitate strong links to the University's strategic planning, quality and review systems to support the University's strategic objectives; support the cyclical planning and improvement of courses through data-driven decision making; and ensure compliance with relevant legislative requirements including responsibilities to international students as defined under the Education Services for Overseas Students (ESOS) Act 2000. 2. Course Quality Cycle As illustrated in Figure 1, QUT s approach to course quality assurance is characterised by the collection and analysis of relevant data, with reference to standards, the design and execution of an improvement process, and the evaluation of the improvement. This involves an ongoing cycle of monitoring and improvement that requires a minimum level of review for all courses and more intense levels of examination for higher risk and underperforming courses. Course quality assurance entails an annual cycle comprising the following three formal reporting components: individual (Individual Course Reports (ICRs) and Individual Unit Reports (IURs)) minimum level for all courses other than those identified as higher risk detailed analysis for higher risk courses consolidated (Consolidated Courses Performance Report (CCPR)) strategic (Strategic Faculty Courses Update (SFCU)) Additional components include faculty corporate reviews external accreditation 3
Figure 1. QUT Course QA Cycle The cyclical nature of Course quality assurance is closely connected to the Curriculum development and approval policy (MOPP C/4.1) and the Evaluation of courses, units, teaching and student experience policy (MOPP C/4.7). Course quality assurance informs the development and improvement of courses and monitors the progress of curriculum changes. Data from ICRs and IURs can be used by staff as part of their personal evaluation strategies to evaluate their courses, units, teaching and student experience since various data sources are aggregated within these reports. The Course Quality Cycle also identifies the measures used to evaluate the effectiveness of those changes. Evidence and trends identified in ICRs and IURs ensure that course evaluation and curriculum change is informed and focused. Quality improvement actions which involve changes to course offerings are summarised in the Strategic Faculty Courses Update which forms the first stage of the course development process. ICRs are considered by Faculty Academic Boards (FABs), with all award courses subject to this minimum level of review. Overarching the process for individual courses, there is a more strategic and higher level process in which each faculty considers summaries of the reports and focuses attention on courses identified as high performing, higher risk, and underperforming. The series of ICRs and associated summaries built up year-by-year is also used by the review panel at the time of the faculty's corporate review to assess the faculty's academic program over the period. ICRs and considerations by the faculties inform the development of the Consolidated Courses Performance Report prepared by the Deputy Vice-Chancellor (Learning and Teaching) which also includes the identification of underperforming courses against predetermined criteria. 4
a. Individual Unit Reports The primary purpose of the Individual Unit Reports (IURs) is to prompt ongoing review of each unit s performance, drawing on quality and viability data through historic and live data. Data from IURs feed into the individual reporting component of the Course quality cycle by assisting Course Coordinators to review their courses based on data specific to units. A secondary purpose of IURs is to provide academic staff with existing sources of data to support the Evaluation of courses, units, teaching and student experience policy (MOPP C/4.7). b. Individual Course Reports The primary purpose of the Individual Course Reports (ICRs) is to prompt an annual check of each course s performance, drawing upon quality, viability and outcomes data. Reporting for courses is a critical and diagnostic analysis focusing on significant trends and issues, action taken and outcomes. A secondary purpose of ICRs is to provide academic staff with existing sources of data to support the Evaluation of courses, units, teaching and student experience policy (MOPP C/4.7). Separate ICRs are required in the following circumstances: where study area As (majors) within a course are substantially distinctive. for a transnational or customised version of a course, with cross-references to the data from the report of the general/onshore offering to provide evidence of equivalence. (Also refer to MOPP C/3.5 Transnational award courses) non-award or CPE programs that articulate to award programs require ICRs. each faculty reports on the performance of the Doctor of Philosophy they manage. (i) Data Data gathered by the University provide valuable information for analysing academic aspects of the course and assessing the adequacy of relevant student services which have an impact on the quality of the students' learning and broader educational experience within the course. Academic staff can also use this data to determine goals for their personal evaluation strategies (see MOPP C/4.7). Faculties are expected to consider the following data sets in the ICR process: the pre-populated data provided in the ICR by the Division of Finance and Resource Planning which includes both current data and multiyear trends across three dimensions: Viability, Quality of Learning Environment and Outcomes; comparison of data against national benchmark data, pre-determined standards (e.g. KPIs) or tends; and additional data gathered by the faculty including internal or external environmental factors, significant issues arising from external evaluations (e.g. corporate reviews, professional accreditation), or further exploration of underperforming factors identified for the course, and benchmarking. High performing and underperforming courses are identified on the front of the ICR top left hand corner by being flagged in either the green (high performing), red zone (underperforming) or shaded grey (missing data) of the sliding scale. The identification of high performing and underperforming courses in the generation of ICRs enables course teams to identify and 5
intensely review such courses and include appropriate actions as part of their responses within relevant ICRs. (ii) Faculty analysis and comments The course coordinator (or other academic leader designated by the executive dean) leads the analysis and reporting for the ICR, together with study area coordinators (where applicable) and course delivery team members 1. Course Coordinators provide their analysis in the comments section of the ICR, which is divided into three sections. Observations, additional data and benchmarks Faculties use the data sets to identify trends and factors that are influencing course performance. This includes: identification of lower and high performance elements within the course; consideration of the consequences of any underperforming elements; outstanding issues arising from the implementation of new curriculum developments (as identified in the relevant curriculum proposal or as required by FAB or UAB); and comments on additional data gathered e.g. exploration of underperforming factors and results of benchmarking activities. Outcomes from previous actions Course Coordinators report on actions from the previous ICR (these actions are automatically pre-populated from the previous year). This reporting includes: Outcomes of actions (performance change if applicable; what s working; what s not); and Status of actions (eg. completed, in progress, refer to new Action Plan commentary). Actions for remainder of year Course Coordinators list new actions resulting from the identification of lower and higher performance elements within the course. This section includes: planned actions to address issues including the development and improvement of curriculum and pedagogical approaches (and incorporating details of what, how, who and when); strategies to share good practice for high performance elements; and additional or ongoing actions resulting from previously identified issues or from course implementation plans. (iii) Criteria for identifying high performing, higher risk and underperforming courses Coursework programs can be classified as high performing, underperforming or higher risk. Levels of risk, performance criteria and weightings are applied according to QUT priorities, national performance indicators and to accommodate contextual factors. These are approved by the Deputy Vice Chancellor (Learning and Teaching). High and underperforming courses are automatically identified drawing upon selected key data across the three clusters of indicators on course viability, quality of the learning environment and course outcomes. Weightings are applied according to QUT priorities, national performance indicators and to accommodate contextual factors. Data for the model is sourced from enrolment, load, QTAC, attrition, unit completions, Course 1 Course delivery teams generally include the unit coordinators responsible for units within the course or major 6
Experience Questionnaire, Graduate Destinations Survey (GDS) and Learning Experience Survey (LEX) data sets as available within QUT Corporate Reporting (QCR). It is expected that metrics and weightings used for underperforming and higher risk courses will evolve over time, and increasingly involve greater comparative benchmarking and draw upon quantitative and qualitative sources of data and alignment with national systems for learning and teaching performance. The performance model in Attachment 1 is the second generation. Criteria for identifying underperforming courses The current model for high performing and underperforming courses was endorsed by the Vice Chancellor s Advisory Committee in 2010 and is outlined in Attachment 1. The model for higher risk courses is provided below. Features of the model include: Management of course performance and issues related to higher risk and underperforming courses as a quality-embedded process linked to individual and consolidated course reports within QUT s Course Quality cycle; Ranking of course performance based upon a discrete set of weighted measures; Capacity to report overall performance summed across the measures and relative to other courses; Capacity to investigate performance in relation to single measures; and Identification of courses if they are new, in teach out mode, have low enrolments, are missing a notable amount of data or have been identified as high or underperforming in consecutive years. The completed ICRs can form the basis of analysis by faculty leaders and FABs to ensure sufficient scrutiny has been undertaken and appropriate actions are planned at the individual course level and as part of faculty plans and priorities. The same courses are reported in an aggregated form as part of the Consolidated Course Performance Report (CCPR) in May of each year for consideration by VCAC and UAB. Criteria for identifying higher risk courses The criteria for identifying higher risk courses are approved by the Deputy Vice-Chancellor (Learning and Teaching). They include the risk factors identified in the TEQSA Risk Framework. Examples of higher risk activity include transnational course delivery, new complex multi-partner corporate courses, underperforming courses with large enrolments (>50) or course teams with very small (1-2) ongoing staff. Higher risk courses are separate from underperforming courses and may require separate treatment such as specific risk management strategies and governance (transnational audits or dedicated project management and governance structures). (iv) Reporting process Faculty-level consideration ICRs are considered through the following processes: The Assistant Deans prepare summary reports, based on the comments provided on individual ICRs by Course Coordinators, for FAB as follows coursework courses the Assistant Dean Learning and Teaching, in consultation with the Assistant Dean (International and Development) for any corporate award courses (Refer Attachment 2 for suggested format) 7
research courses, one-year bachelor honours and all professional doctorates the Assistant Dean Research, in consultation with the Assistant Dean (International and Development) for any corporate award courses double degrees the ICR is considered at each partner faculty at the same time as the related single degree course. The faculty with the first named degree is responsible for completing the ICR and initiating contact with the partner faculty (unless otherwise negotiated). FABs may choose to either consider all ICRs for the faculty or consider selected ICRs as recommended by the summary reports (high performing and higher risk courses). Faculties may choose to submit ICRs or Assistant Dean summaries for consideration by other relevant committees/advisory groups (faculty learning and teaching committees, faculty/school advisory groups on curriculum matters). ICRs for corporate award courses are also considered by relevant faculty governance structures. For some courses, these may include different committees than for other award courses in which case the default responsibility for their presentation is with the Assistant Dean (International and Development) or equivalent. Refer to the Protocols for Customised Education. FABs approve ICRs for submission (together with the relevant FAB minute extract) to the Deputy Vice Chancellor (Learning and Teaching) for coursework courses or the Deputy Vice-Chancellor (Research and Commercialisation) for research courses. University-level consideration The Learning and Teaching Unit consolidates information into the Consolidated Course Performance Report and provides University Academic Board (UAB) with information on higher risk courses. ICR Summary Reports and ICRs for research courses are considered by the Deputy Vice- Chancellor (Research and Commercialisation) for consideration by Research Degrees Committee. (v) Timelines Individual Course Reports (ICR) are released after the data snapshot in March. Course Coordinators and Course Teams complete comments on their ICRs during the month of April. Courses identified as underperforming, as shown on their ICR or as higher risk as identified by the Assistant Dean Teaching Learning, may include in their team comments the analysis of additional data to track the results of actions previously taken to improve the course or may include a proposal to discontinue the course. QUT defines minimum student intakes for both undergraduate and postgraduate courses in the criteria used to identify underperforming courses (a minimum of 10 enrolments per 48 credit points of study). The Vice-Chancellor may, in accordance with minimum intakes, decide not to have an intake into a course if applications are insufficient to achieve viability. The table below shows the critical dates and reporting pathways. 8
Jan Feb Mar Apr May Individual (IUR & ICR): IUR release 2013 data Update ICR Performance Model 2013 version ICR team comments open data QCR identification of Underperforming Courses AD(T&L) identification of Higher Risk Activities 02-Jan-13 04-Mar-13 15-Apr-13 15-Apr-13 15-Apr-13 Course teams enter comments and action plans 15-Apr-13 13-May-13 ICR User training 15-Apr-13 13-May-13 ICR closure of comments AD(T&L) summary of courses including Underperforming and Higher risk activities to FAB 15-Apr-13 13-May-13 29-May-13 FAB approval 15-Apr-13 29-May-13 AD(T&L) summary submitted to DVC(LT) 29-May-13 DVC(LT) approves amendments to the course quality assurance proecedures. Areas of responsibility: QUT Corporate Reporting Course team Faculties Learning & Teaching Unit c. Strategic Faculty Courses Update and Performance and risk checkpoints Actions from faculties in response to quality data may include curriculum developments that are managed through the curriculum development and approval process. Each faculty provides a summary of these proposed curriculum developments in the Strategic Faculty Courses Update (SFCU), which is prepared as part of the strategic planning and review cycle each year and provides an overview of the anticipated strategic direction of the faculty s academic programs. The executive dean of faculty is responsible for preparation of the SFCU (although this responsibility is usually delegated to the Assistant Dean, Learning and Teaching). The SFCU is therefore both a summary of major actions resulting from the quality assurance cycle, as well as a planning document and the initiating step for curriculum development and approval processes. The SFCU includes: a brief analysis of the faculty s strategic position relating to courses; and a summary of planned action including plans for curriculum development, informed by the faculty s consideration of its ICRs, the CCPR and underperforming courses and higher risk activities. The SFCU is intended to provide information to University stakeholders, and faculties are encouraged to use their SFCU to initiate discussions with Divisions and other stakeholders about their planned curriculum developments for the following year. However, it is expected that additional course proposals that were not included in the original planning document may arise opportunistically during the year. (i) Reporting process The SFCU is endorsed by FAB and forwarded to FRP as part of the Strategic Planning Process. The Learning and Teaching Unit prepares a consolidated report which is considered by: VCAC for consideration of viability and risk matters 9
UTLC for consideration of quality themes. Courses identified as higher risk activities, having been noted in April, receive further scrutiny with a review of progress against action plans occurring at the performance and risk checkpoints in September/October to track improvements. Commentary regarding the future direction of these courses is then included in the Strategic Faculty Courses Update. (ii) Timelines The diagram below shows the critical dates and reporting pathways. Sep Oct Nov Strategic Faculty Courses Update Strategic Faculty Courses Update Early Sept Late Oct LTU consolidated report on Faculty SFCUs 2014 VCAC consideration of viability and risk matters 2014 ULTC consideration of quality themes 2014 DVC(LT) approves amendments to the course quality assurance proecedures. Areas of responsibility: QUT Corporate Reporting Course team Faculties Learning & Teaching Unit d. Consolidated Courses Performance Report The primary purpose of the Consolidated Courses Performance Report (CCPR) is to provide an overall evaluation of the University s course performance and identify high performing and underperforming courses. The CCPR is prepared by the Deputy Vice- Chancellor (Learning and Teaching) in consultation with the Deputy Vice-Chancellor (Research and Commercialisation) and draws on University-wide quality data and ICRs. The CCPR is released in May each year. The CCPR amalgamates data and analysis and includes: university level environmental scan of course-related areas; consolidation and reference to data and analysis across all courses, and tracking of trends on key learning and teaching issues and priorities; select analysis against course types for example, undergraduate, postgraduate, higher degrees and research, domestic, international, corporate award and transnational; identification of high performing, higher risk and underperforming courses (at study area A levels) according to predetermined criteria using three broad categories: viability, quality of learning environment, and outcomes; relevant benchmarking against standards and national data, where available; closing the loop on previous actions to improve course performance; and links and references to associated analysis and data sets produced since the previous report. (i) Reporting process The CCPR is considered by the following committees: VCAC for consideration of viability and risk matters UAB for consideration of academic matters Council consideration of strategic performance 10
UTLC for consideration of quality themes. (ii) Timelines The diagram below shows the critical dates and reporting pathways. Mar Apr May Jun Jul Aug Consolidated Courses Performance Report (CCPR) LTU identification of Themes for CCPR Full dataset for QUT ICRs available from QCR QCR preparation of CCPR tables Director of Reporting & Analysis preparation of CCPR commentary LTU CCPR preparation of Theme pages ULTC consideration of quality themes VCAC consideration of viability and risk matters UAB consideration of academic matters Council consideration of strategic performance 16-Apr-13 16-Apr-13 30-Apr-13 14-May-13 28-May-13 25-Jun-13 03-Jul-13 19-Jul-13 07-Aug-13 DVC(LT) approves amendments to the course quality assurance proecedures. Areas of responsibility: QUT Corporate Reporting Course team Faculties Learning & Teaching Unit 11
3. Additional components of quality assurance a. Faculty corporate reviews Corporate reviews form a core part of the University's planning and quality assurance framework (see MOPP A/2.3) The corporate review of a faculty includes an external assessment, from a strategic perspective, of the overall suite of courses offered. Review panels consider the ICR summary reports prepared for FABs and are provided with access to the full set of ICRs submitted to the Learning and Teaching Unit, including status reports of underperforming courses. Review panels are not expected to incorporate detailed scrutiny of all individual courses, but will take an evidence-based approach to the following: appropriateness of the mix of courses offered by the faculty; range and level of courses offered; effectiveness of quality assurance processes used by the faculty for assuring standards and facilitation improvement, including outcomes since the previous review and deliberation by internal and external committees; graduate and learning and teaching outcomes from the range of courses; equivalence experience and outcomes of transnational or corporate award courses to domestic offerings; and rigor of external review and moderation process. b. External accreditation Course Quality Assurance includes periodic external review as part of curriculum approval and review cycles between corporate review. These activities may include: professional accreditation; external accreditation; commissioned reviews (discipline, theme and course levels); and moderation (learning outcomes, assessment). External review and accreditation contributes to Course Quality Assurance through the independent validation of professionally recognised standards and facilitates benchmarking. Faculties with courses that are not covered by professional accreditation are expected to include alternative approaches to external review to ensure the highest possible quality of their course offerings. 4. Roles and responsibilities University officers and faculty / school committees Course Coordinators (or other designated academic leaders nominated by the executive dean) are responsible for their course and lead the course team in the analysis of the ICR data and development of the comments and action plan of the ICR throughout each year review the action plans of higher risk activities in consultation with the Assistant Dean Learning and Teaching in preparation for the SFCU submission in October Assistant Deans, Learning and Teaching are responsible for all coursework courses within their faculty and 12
prepare a thematic overview and summary of ICRs including underperforming and higher risk activities in April/May for submission to FAB for all coursework award courses (including corporate award courses) prepare a report on the status of courses identified as higher risk ifor inclusion in the SFCU in October develop the SFCU as the Executive Dean s nominee develop and oversee all associated QA reporting processes for coursework courses within the faculty including ICRs, SFCU and underperforming and higher risk courses and submissions to Learning and Teaching Unit consider implications of course performance issues for faculty learning and teaching plans and priorities follow up on the appropriateness of Course Coordinators comments submitted in the ICRs Assistant Deans, Research are responsible for all research courses within their faculty and develop and oversee all associated QA reporting processes for research courses within the faculty including ICRs, SFCU and underperforming courses and submissions to Division of Research and Commercialisation consider implications for faculty research plans and priorities follow up on the appropriateness of Course Coordinators comments submitted in the ICRs prepare and present an ICR Summary Report to FAB for all research programs (including honours by research) Assistant Deans, International and Development (or equivalent) contribute to the development of summaries for corporate award courses follow up and report on action plans to relevant governance and management structures present ICRs for corporate award courses at relevant faculty governance structures for customised education Faculty Academic Boards endorse faculty submissions on ICRs, higher risk courses and the SFCU forward relevant minute extracts of ICRs to the Learning and Teaching Unit and Division of Research and Commercialisation monitor progress of underperforming and higher risk courses consider implications for academic planning including curriculum development Heads of School oversee effective quality processes within their school contribute to the review of courses within their school monitor the delivery of action plans by their staff monitor underperforming courses Executive Deans of Faculty oversee effective quality processes close the loop on previous actions prepare and submit SFCU Deputy Vice-Chancellor (Learning and Teaching) 13
oversees course quality review system prepares CCPR monitors underperforming and higher risk courses facilitates the dissemination of good practices considers implications for planning and review Deputy Vice-Chancellor (Research and Commercialisation) monitors underperforming and higher risk research courses facilitates dissemination of good practices considers implications for planning and review Executive Director, Finance Resources and Planning is responsible for the data collection and reporting within the ICRs and the CCPR and assures quality and timeliness of data supports reporting and analysis of data provides training in the use of tools University committees University Learning and Teaching Committee considers quality themes arising from the CCPR considers quality themes arising from the SFCUs provides advice on implications for planning and review Vice-Chancellor s Advisory Committee considers viability and risk matters and endorses the CCPR considers the SFCUs considers viability and risk matters relating to underperforming courses considers implications for planning and review University Academic Board considers academic matters arising from ICRs considers academic matters and approves the CCPR monitors academic matters arising from underperforming courses and follow-up status update considers implications for planning and review provides Council with: a strategic analysis of the University's performance in courses and learning and teaching (drawn from the CCPR), and advice on significant performance issues in learning and teaching. 14
Attachment 1: Model for identifying high performing and underperforming courses The proposed approach to identifying high performing and underperforming courses has been developed in accordance with several key principles. In particular, it has been identified that an effective model: Requires robust quantitative data sources that are consistent across courses; Allows roll ups and drill downs of courses within ICRs and CCPR, at faculty and university levels; Uses consolidated data sets that reflect the most comprehensive course quality picture possible, whilst still being easy to understand and practical to manage; Utilises a selective set of weighted indicators that is consistently able to identify courses with significant quality risks at one end and courses that are extremely well performing at the other end; and Has selected indicators that capture course performance benchmarked against QUT KPIs and national benchmarks wherever possible. The measurement of course performance has been designed around a set of rules: There are 12 performance indicators selected for use in undergraduate courses and 11 for post graduate courses. Each has been designated as a requisite, primary or secondary indicator. Performance scores can reflect annual change against the previous score, performance against a target set by the university, or performance relative to a national average. For each indicator, a cut-off score has been nominated for high performance and under performance. Cross-institutional courses (like Languages, offered collaboratively with other Brisbane universities), non-award and other award offerings and HDR courses undergoing other forms of QA, have been excluded from this model. 15
Attachment 1: Categories, Measures and Proposed Criteria / Cut-offs for the Course Performance Model Category Measure High Performing Underperforming Requisite 1 Courses with complete datasets Requisite 2 Courses with more than 10 enrolments per 48 Credit Points PM1 OP Cut Off An increase of greater than one (Applies to UG equals positive score (+1) Courses only) Primary Measures Secondary Measures PM2 First Preferences to Offers (Applies to UG Courses only) Commencing Attrition above 3:1 equals positive score (+1) PM3 less than 10% equals positive score (+1) PM4 Attrition less than 10% equals positive score (+1) PM5 Student Progress no positive score is awarded Rate (SPR) PM6 CEQ Good Teaching PM7 PM8 CEQ Overall Satisfaction EFTSL Change (Applies to PG Courses only) greater than 15 points above the national BFOE average equals positive score (+1) greater than 15 points above the national BFOE average equals positive score (+1) 10% increase in EFTSL from the previous year equals positive score (+1) SM1 CEQ Generic Skills greater than 15 points above the national BFOE average equals positive score (+0.5) SM2 GDS FT Employment greater than 10% above the national BFOE average equals positive score (+0.5) or 100% total SM3 GDS FT Study greater than 10% above the national average equals positive score (+0.5) or 100% total decrease of greater than one equals negative score (-1) below 0.75:1 equals negative score (-1) greater than 20% equals negative score (-1) greater than 20% equals negative score (-1) less than 80% equals negative score (-1) less than 15 points below the national BFOE average equals negative score (-1) less than 15 points below the national BFOE average equals negative score (-1) 10% decrease equals negative score (-1) less than 15 points below the national BFOE average equals negative score (-0.5) less than 10% points below the national BFOE average equals negative score (-0.5) less than 10% points below the national BFOE average equals negative score (-0.5) The scale potentially runs from -9.5 underperforming to +8.5 high performing for undergraduate courses and from 8.5 to +7.5 for post graduate courses. If the course scores -2 or below, it is identified as underperforming. If +2 or above, it is identified as high performing. For 2013, as QUT transitions to our new evaluations framework and due to the discontinuation of LEX, the 2012 SM2 and SM5 item have been removed from the performance model. Note: AUSSE appears on the ICR for those courses surveyed and receives a positive indicator where the score was greater than the AustAsian Average + 5 and a negative indicator where the score was less than the AustAsian Average 5. However this is not included in the calculation of the Performance score as not all courses conduct the AUSSE. Attachment 2: Suggested format for ICR Summary Report 16
Total length of report should be 2 pages with attachments Common themes Considered whether there are any common themes (strengths or weaknesses) for example: across the faculty, school(s), course(s), study area(s) with respect to particular types/groups of courses (double degrees) or units (first year, WIL, capstone) with respect to the five T&L priorities (retention, curriculum design, real world learning, assessment, accessibility) across three dimensions (viability, learning environment and outcomes) missing or incomplete data sets (new courses, small enrolment courses, double degrees) References to aid analysis: Faculty level reports Consolidated report (quantitative data) Summary of comments from across ICRs Sand pit to provide opportunity to cut data according to faculty needs (ie across courses, study areas, schools, teams) Individual Course Performance Reports Underperforming courses (where identified) Summarise those courses identified as Underperformings. Review history/trend data to determine where they have been previously identified as underperforming. If previously identified, what is the status of the action plan, is it on track or require adjustment, do you have sufficient data? If new, is the action plan realistic Is six monthly monitoring required? Note: this information/analysis should directly flow into the formal reporting requirement on underperforming courses (summary required by the end of March) along with the reporting in the Consolidated Courses Performance Report. High performing courses Summarise those courses that have been identified as high performing. Identify whether there are areas of good practice that should be shared more widely, particularly with those responsible for underperforming courses. Outline dissemination strategies Higher Risk Activities (where relevant) Summarise those courses identified as higher risk activities. Examples of higher risk activities include transnational course delivery, new complex multi- partner corporate programs, underperforming courses with large enrolments 17
(>50) or course teams with very small (1-2) ongoing staff. Higher risk courses can be separate to underperforming courses and may require separate treatment such as specific risk management strategies and governance (eg. transnational audits or dedicated project management and governance structures). Review history/trend data to determine where they have been previously identified as underperforming or high risk. If previously identified, what is the status of the action plan, is it on track or require adjustment, do you have sufficient data? If new, is the action plan realistic Is six monthly monitoring sufficient? Benchmarking or other data Outline plans or results of work undertaken by the faculty to further investigate performance. where ICR is insufficient (eg. small courses with low returns) qualitative data (CEQ Query; LEX) national and international data relating to AUSSE (ie ATN, G08, NSSE) Where faculty s FOE s sits against national benchmarks (CEQ, GDS, Attrition) Implications for faculty plans Outline the impact of the above on faculty plans or major curriculum actions, such as: Faculty s learning and teaching plan o o Confirm whether strategies and priorities still current or on track Outline strategies/tactics that require monitoring or adjustment (perhaps earmark for the next formal iteration of the plan Strategic Faculty Courses Update o Outline any new curriculum developments that need to be considered for 2010 Professional accreditation/corporate or discipline reviews o o Outline whether there are any issues that need to be considered or addressed with respect to professional accreditation Include consideration for corporate or discipline self reviews or future action plans Adjustment of priorities or performance indicators for schools/disciplines/courses Professional development follow up Additional plans for review or performance monitoring Closing the loop from the previous ICR Summary Report Outline any actions or issues that were raised in the previous ICR Summary Report that have not been covered by the above. ICR Process Outline areas for improvement for the faculty or university with respect to the reporting processes. Target area: Double degrees. Possible attachments 1. Faculty summary data (or sandpit derivative) 18
2. Higher Risk Activities (all ICRs) 3. Underperforming courses (max top 5 ICRs) 4. High Performing courses (max top 5 ICRs 19