Scope and level of evaluations

Transcription

1 are mostly positive, and the audit teams have often been looked upon as consultants or advisory groups to the university managements. In a few cases there have been negative reactions. An important aspect has the learning process for both institutions and, not least, the visiting teams, which has been commented on by several of those involved. As a consequence of the overall positive results a second cycle of quality audits has recently been decided on by the Board of the Agency. In the main, they will follow the same principles as the first cycle, but draw on the experiences made during those years. It is clear, however, that evaluation efforts by the Agency will now more clearly emphasize programme and discipline assessments, following indications from Government and students that it is time to stress the quality as such of the education offered at universities and colleges. United Kingdom Prepared by The Quality Assurance Agency for Higher Education. The Quality Assurance Agency for Higher Education (QAA) is a limited company and a registered charity. It was established in 1997 by the higher education institutions in the UK through their representative bodies. Four members of the Board of Directors are nominated by the representative bodies, four are nominated by the Government Agencies that fund higher education and six are independent. The independent members, one of whom is the Chairman, are representatives of those sectors of industrial, commercial and professional employment that recruit students on graduation. The Agency is staffed by a Chief Executive, four Directors and 80 professional and administrative staff. The QAA is funded from two sources: by subscriptions levied from UK higher education institutions, the amount varying according to the number of full-time equivalent students registered at the subscribing institution, and through service contracts with the English and Welsh Funding Councils. In 1997/98 the total funding available to the Agency was #8 million. The mission of the Agency is to promote public confidence that quality of provision and standards of awards in higher education are being safe-guarded and enhanced. It will do this by: working with higher education institutions to promote and support continuous improvement in the quality and standards of provision; providing clear and accurate information to students, employers and others about the quality and standards of higher education provision; working with institutions to develop and manage the qualifications framework; advising the Government on the grant of degree awarding powers and university title; facilitating the development of benchmark information to guide subject standards; promulgating codes of practice and examples of good practice; operating programmes of review of performance at institutional and programme levels. Scope and level of evaluations The scope of the Agency's work at institutional level covers all UK higher education subscribing institutions currently 180 in total. All publicly funded higher education in England and Wales and Northern Ireland is covered by the subject review activities carried out by the Agency.

2 Focus of evaluations The Agency's review activities operate at both institutional and programme level. Institutional level Since 1991 all UK universities and colleges have been audited and a fresh round of such audits now know as continuation audits' began in the 1997/98 academic session. Continuation audit looks at how individual institutions discharge their obligations and responsibilities for the academic standards and quality of their programmes and awards and at the evidence they themselves are relying on for this purpose. There are also audits of institutions' collaborative provision, that is, programmes delivered in partnership with institutions in other sectors of education or overseas. Programme level Whereas audit looks at the way institutions manage their responsibilities for the quality of programmes and awards offered in their names, subject review (sometimes known as teaching quality assessment or TQA) looks at the quality of teaching and learning in specific subjects or programme areas within institutions. It is concerned with students' learning experience and students' achievement at the subject level. The Agency inherited the current cycle of subject reviews from the Funding Councils and cycle will be completed in 1998 in Wales and is scheduled to be completed in 2001 in England. Objectives of evaluation Accountability lies at the heart of the mission of the Agency but this accountability goes beyond demonstrating value for public money. The purposes of external quality assurance processes include: to assist institutions in enhancing the quality of their provision; to promote public confidence, at home and abroad, in higher education and the standards of awards; to enable the funding councils to fulfil their statutory responsibilities; to generate reliable public information that is helpful to potential students, employers, parents, government, funding councils and the institutions themselves; to ensure that there is clarity and transparency about the purposes of programmes and the meaning of awards; to provide a measure of accountability for the resources provided by the public purse and individuals to fund institutions. The main elements of evaluation procedures Institutional evaluation: the audit method in summary The purpose of audit The purpose of audit is to establish the extent to which institutions are discharging effectively their responsibilities for the standards of awards granted in their name, and for the quality of education provided to enable students to attain those standards. The objects of audit Audit asks each institution the following questions. Can you tell us how you know whether you are discharging effectively your responsibility for the standard of each award granted in your name and for the

3 quality of education provided by you to enable students to attain that standard? Can you convince us that the evidence that you are relying on for this purpose is sufficient, valid and reliable? The audit team Audit is a process of peer review. Audit teams comprise three auditors, usually senior academics from higher education institutions in the UK, and an audit secretary, generally a career administrator with experience of servicing academic committees in higher education institutions, or with related experience. An Assistant Director of the Agency manages and coordinates the whole of the audit process. The evidence used The key documentary evidence considered by an audit team is an analytical account produced by an institution (a self assessment document) at the start of the audit process. This document describes and comments on the means used by the institution to test whether it is discharging its responsibility for standards and quality. The account will refer to an accompanying list of the evidence used by the institution itself to test the effectiveness of the discharge of its responsibilities. Any of the documents listed can be requested by audit teams, along with any other documents referred to during the course of the team's visit to an institution. In addition, audit teams have available to them other evidence, including Funding Council teaching quality assessment reports and if available accreditation reports which offer valuable, external perspectives. Finally, audit teams visit institutions and obtain oral evidence from staff and students. The audit visit Each audit involves a visit to the institution. This normally lasts for up to five days, of which two or three will be taken up by discussions with small groups of staff and students. The remainder of the time is set aside for reading documentary evidence placed in a base room' by the institution, and by discussions within the audit team itself. The team stays together throughout an audit visit, and it is the team's considered conclusions and their combined drafting skills which form the basis and create the overall tone of the audit report. Length of the audit process For auditors, the process extends over a period of about five months, from the point when an audit team receives the Analytical Account and supporting documentation (normally at least two months before a visit) to the delivery of the draft report for comment by the institution (normally no more than eight weeks after the visit). The outcomes of audit An audit team's findings are summarized in a published report which reviews the evidence obtained during the course of the audit and offers a view on: the rigour with which an institution tests whether it is discharging effectively its corporate responsibility for standards and quality; the sufficiency, validity and reliability of the evidence used by an institution to test whether it is discharging effectively this responsibility;

4 the level of confidence which an informed observer might reasonably place in the effectiveness of the processes and procedures used by an institution in discharging its responsibilities. Audit reports end with a general summary of the audit team's findings as well as a list of the policies and practices which the team has observed and which it considers worthy of commendation. Finally, the report lists areas where the audit team suggest that further work be done. Subject review The subject review (teaching quality assessment) method evaluates the quality of educational provision within a subject area as defined by a unit of review. It is focused, at the level of the subject, on the quality of the student learning experience and student achievement. Taught programmes of study at all levels are reviewed. Peer review Reviews are carried out by a team of registered subject specialist reviewers, led by a review chair. It is the subject specialist reviewers main responsibility is to collect evidence and to make judgments on the quality of education provided. Subject specialist reviewers are drawn mainly from the higher education sector, although nominations and applications from industry, commerce and the professions are actively sought. Subject specialist reviewers are trained by the QAA in conjunction with the Universities and Colleges Staff development Agency (UCOSDA) before their first review visit. Each newly appointed review chair is provided with training in managing visits and other relevant skills and attends a subject specialist reviewer training course led by UCOSDA. Self Assessment The self assessment is an evaluation by the subject provider (institution) of the quality of the students learning experience and student achievement, measured against the aims and objectives that the subject provider sets for the education of its students in that subject. This evaluation should be organised within the structure given by the aspects of provision (see below) and should be supported by evidence. A self assessment should discuss both strengths and weaknesses in the provision. Guidance on the preparation of self assessments is provided by the Agency in a Handbook. The self assessment is the primary guide for the reviewers in setting the priorities and planning the programme for the review. It is analysed prior to the review visit. Review against the Subject Provider's aims and objectives A statement of the subject providers aims and objectives is included in the self assessment; it provides the framework for the review and is published in the subject review report. The statement of aims and objectives should be sufficiently clear to enable a review visit to be planned and undertaken. Reviewers evaluate the extent to which the student learning environment in each of the aspects of provision contribute to meeting the objectives set by the subject provider. Reviewers also evaluate whether the objectives set, and the level of attainment of those objectives allow the aims set by the subject provider to be met. The subject review visit The purpose of the review visit is to gather, consider and test the evidence of the quality of education in the light of the subject provider's aims and objectives and to establish a graded profile and an overall judgement on the quality of that provision. A review visit normally lasts three and a half days. A common range of activities is

5 undertaken during visits, including the observation of teaching and learning, although the precise order and balance may be adapted to reflect the nature of the provision. The graded profile The graded profile shows the extent to which the students learning experience and student achievement demonstrate that the aims and objectives set by the provider are being met. The profile is created by applying a grade to each of the six aspects of provision. The assignment of grades is a matter for the professional judgement of the review team, drawing on the evidence from both the self assessment and the review visit. Include description of grading? The overall summative judgement The overall summative judgement is derived from the profile. Each aspect has equal weight. The judgements are given to the institution in oral feedback by the review team at the end of the visit. The subject review report A subject review report is published after each review visit and is the main documented outcome of the review process. Reports concentrate on the judgements made using the specified criteria and the evidence which supported them, in relation to the subject provider's aims and objectives. The reports are published on the Internet: the current web-site address is Subject overview reports On completion of the subject reviews for 1998 to 2000, the Agency will evaluate the overall quality of the higher education provision within the subject areas reviewed and publish subject overview reports that are designed to promote best practice and enhance quality. Each subject overview report includes a list of providers in the subject area and the graded profiles achieved by these providers. The main aspects covered in the evaluation Institutional quality audit In order to structure the audit enquiry to enable them to prepare their report, audit teams concentrate on four main areas of scrutiny: the institutional quality strategy (that is the way in which the institution as a whole approaches matters of quality and standards) the academic standards of programmes and awards; the learning infrastructure internal and external communications. The Agency believes that the way these areas are tackled by an institution is critical to the effective discharge of its responsibility for quality and standards. The audit enquiry extends to the full range of an institution's academic provision. Subject review: aspects of provision The aspects of provision provide a common structure for each stage of the subject review process: the self assessment, the subject review visit, the judgments and the subject review report. The six aspects of provision are: Curriculum design, content and organization Teaching, learning and assessment Student progression and achievement Student support and guidance Learning resources Quality management and enhancement It is intended that the aspects of provision should be sufficiently broad and inclusive to enable providers and reviewers to give full weight to the particular features of the provision in different contexts.

6 Expected major developments The report of the National Committee of Inquiry into Higher Education (NICHE) chaired by Sir Ron Dearing - Higher Education in the Learning Society - and the report of the NICHE's Scottish Committee chaired by Sir Ron Garrick were published in July The reports outline a new model for assuring quality and standards in UK higher education that builds on the experience of previous models but which is broader in scope and provides an integrated model that takes account of the needs of all stakeholders. They proposed that the QAA should work with higher education institutions to produce a national framework for higher education qualifications in which all higher education awards would have a consistent terminology. In each subject area, the academic community would develop benchmark standards for the achievements expected at various levels of award. The external examiner system (an informal mechanism operated by higher education institutions) could be strengthened by creating a UK wide pool of external examiners recognized by the QAA. The NICHE recommended that underpinning the external quality assurance process would be a series of codes of practice covering all quality assurance matters which institutions would be required to abide by. Adherence to the codes would be checked by a five-yearly institutional review. The Agency is currently requesting the views of the higher education sector and of stakeholders on its proposals in relation to: developing qualifications frameworks in the UK developing a template for programme specification developing benchmark information on subject threshold standards defining the subject areas for bench marking work; strengthening the role of the external examiner system. The revised approach to quality and standards offers the potential for a reduction in the burden of external scrutiny on institutions, to be achieved by: a clearer specification of what is expected of institutions; a focus on outcomes rather than a detailed investigation of teaching and learning processes; combining the information needs and review processes of different external agencies. The proposed timetable for change is that the new model for external quality assurance will be in place by the year Political and methodological developments See the paragraph on expected developments above. The Agency advises the Government on the granting of degree awarding powers for taught courses and research degrees and also on the granting of university title. A review of the process and criteria for degree awarding powers is currently underway. Scotland The Scottish evaluation cycle has been finalised in the Summer of The organisation, scope and focus of the future evaluation activities are currently being discussed. 4. EXPECTED DEVELOPMENTS IN EVALUATION AND QUALITY ASSURANCE OF HIGHER EDUCATION The role of evaluation agencies and the purposes of evaluation