Impact of League Tables and Rankings on Institutional Diversity Ellen Hazelkorn Higher Education Policy Research Unit (HEPRU) @ CSER Director, and Dean of the Faculty of Applied Arts Dublin Institute of Technology Higher Education in the 21 st Century Diversity of Missions Dublin, June 2007
'The University itself is ranked among the top UK universities for the quality of its teaching' You should hold a degree from a Times top 100 university ranked at no 33 or higher League Tables are dangerous, often ill-informed but difficult to influence and most definitely here to stay!
Themes 1. Rankings: A Challenge to HEIs? 2. Position and Reputation 3. Responding to Rankings 4. Enhancing Global Competitiveness and Increasing Stratification
1. A Challenge to HEIs?
Difficulties with League Tables Technical and Methodological Difficulties Indicators as proxies for quality? Quality and appropriateness of the metrics Usefulness of the results as consumer information Rater bias? Halo effect? Reputational ranking? Quality and appropriateness of the information Comparability of complex institutions One-size-fits-all? Diversity of missions, complex organisations Matthew effect? Influences on institutional decision-making and academic behaviour Helping set strategic goals or encouraging HEIs to become what is measured?
Challenges for HE and HEIs Are League Tables and Ranking Systems influencing and informing institutional decision-making? strategy and mission institutional priorities academic and research resource allocation recruitment and marketing Do HEIs monitor the performance of peer institutions? Do League Tables and Ranking Systems influence collaboration or partnerships? Do League Tables and Ranking Systems influence the views or decisions of key stakeholders? Are League Tables and Ranking Systems influencing broader higher education objectives and priorities? Who should undertake ranking and which metrics should be used?
International Study Conducted in association with IMHE (OECD) and IAU using their membership lists. Email questionnaires sent to leaders/senior administrators in June-September 2006. 639 questionnaires sent, with some unquantifiable snowballing 202 replies received 31.6% response rate
Respondent Profile (N=202) Age: 36% post 1970 24% 1945-1969 40% pre 1945 83% publicly funded Institutional type 30.4% teaching intensive 19.3% research informed 29.2% research intensive
Global Distribution 41 countries, N=155 Australia Austria Belguim Canada China Denmark Finland Germany Greece Iceland India Iran Iraq Ireland Italy Japan Jordan Keny a Lebanon Lithuania Mexico Mongolia Netherlands N Cyprus Norway Paraguay Philippines Portugal Rw anda Saudi Arabia Serbia Slov enia South Africa Spain Sri Lanka Sw eden Sw itzerland Thailand Turkey UK USA
2. Position and Reputation
Popularity and Purpose of Ranking Use of national rankings on the rise, but worldwide rankings have wider penetration. Over 70% respondents identified providing comparative information as the primary purpose of LTRS However, there is a differentiation between the target audience and user of such surveys: Target audience: students and public opinion User: public opinion, government, parents and industry
Ranking Status Significant gap between current and preferred rank 93% and 82%, respectively, want to improve their national or international ranking. 58% respondents not happy with current institutional ranking Current ranking: 3% of all respondents are nationally ranked 1 st in their country, but 12% want to be so ranked; No respondents are internationally ranked 1st, but 3% want to be so ranked 70% of all respondents wish to be in top 10% nationally, and 71% want to be in top 25% internationally.
Maintaining Position and Reputation Rankings play a critical role in enabling/facilitating HEIs to maintain and build institutional position and reputation. While answers dependent upon happiness with position, almost 50% use their institutional position for publicity purposes: press releases, official presentations, website. 56% have a formal internal mechanism for reviewing their 56% by the Vice Chancellor, President or Rector 14% by the Governing Authority
Peer-benchmarking Over 40% of respondents said they considered an HEI s rank prior to entering into discussions about: international collaborations academic programmes research student exchanges 57% said they thought LTRS were influencing willingness of other HEIs to partner with them. 34% said LTRS were influencing willingness of other HEIs to support their institution s members of academic/professional organisations.
Influence on Key Stakeholders Examples Benefactor Collaborators Current Faculty Employers Funding Agencies Future Faculty Government Industry Parents Partnerships Students Depends on the rank They feel reassured supporting us Provides international comparators Depends on the rank Good for reputation we feel an improvement Increases awareness about the importance of publishing Easier to induce improvement with the department head whose rankings are declining They feel reassured Those not open to us become more receptive Can be confusing Impact on small part of indicators Have less pretexts to deny funding Recruitment easier with good reputation May believe simplistic picture Local government included to spend additional money for an excellent university Depends on the rank: good for reputation vs. less interest Particularly in an international market where status and prestige are considered in decision-making Good for reputation at international level High profile students apply to high profile universities Influence at the margins
3. Responding to Rankings
Actions Arising (1) 63% respondents have taken strategic, organisational, managerial or academic actions in response to the results Of those, Overwhelming majority took either strategic or academic decisions and actions Only 8% respondents indicated they had taken no action
Actions Arising (2) Examples Strategy Organisation Management Academic Indicators underlying rankings are explicit part of target agreements between rector and faculties Have become part of a SWOT analysis Organise benchmarking exercises New section established to deal with indicator improvements and monitor rankings Reorganisation of structure Have organised investigation team Rector enforces the serious and precise processing of ranking as well as control of the relevant indicators Development of better management tools Improve teaching and learning New academic programmes Increase English language programmes More scholarships and staff appointments
Impact on Higher Education True % False % Favour Established Universities 83 17 Establish Hierarchy of HEIs 81 19 Open to Distortion and Inaccuracies 82 18 Provide Comparative Information 74 26 Emphasize Research Strengths 65 35 Help HEIs Set Goals for Strategic Planning 65 35 Provide Assessment of HEI Performance 52 48 Promote Accountability 48 52 Can Make or Break an HEI s Reputation 42 58 Provide Assessment of HE Quality 41 59 Promote Institutional Diversity 38 62 Enable HEIs to Identify True Peers 33 67 Encourage FAIR Competition 25 75 Provide Full Overview of an HEI 11 89
Ideal League Tables Should give fair and unbiased picture of strengths and weaknesses Provide student choice for a programme and institution Provide accountability and enhance quality Ideal metrics are: Teaching quality Employment Student-staff ratio Research, e.g. publications and income Should be developed by independent research organisations, accreditation agencies or international organisations. Favour institutional reviews (41%) rather than at programme (29%) or departmental level (30%).
3. Enhancing Competitiveness and Increasing Stratification
Impact (1) Despite methodological concerns strong perception that: Benefits/advantages flow from high ranking; Rankings taking on QA function but with different definitions of quality (Usher and Savino, 2007) Influence goes beyond traditional student audience Growing influence on public opinion, government and industry; Influence policymaking, e.g. classification of institutions, allocation of research funding, accreditation; University administrators most engaged and obsessively implicated.
Impact (2) HEIs taking results very seriously, and making changes Embedding league tables within strategic decision-making analysis; Making structural and organisational changes: Shift resources from teaching to research (Bennis & O Toole, Chronicle HE, 2005); Publicity and marketing. Institutions behaving rationally becoming what is measured.
Other Impact Evidence (1) Employers favour graduates from more highly ranked HEIs (UK) (University of Sussex, 2006) State appropriations per student in public colleges are responsive to rankings (US) (Zhe Jin, 2007) Influence on marketing and recruitment: High rankings rise in applications (NY Times, 2007) Recruiting students who will be assets in terms of maintaining/enhancing rank (Clarke, 2007) HEIs making extensive investments to influence student input metric (Brewer et al, 2002)
Other Impact Evidence (2) Almost all universities chosen for Deutsche Telekom professorial chairs used rankings as evidence of research performance (Spiewak, 2005) Scholarships for study abroad restricted to students admitted to highly ranked universities e.g. Mongolia, Qatar (Salmi & Saroyan, 2007) Arizona Board of Regents approved a contract this year to give president of Arizona State University a $10,000 bonus if institution's U.S. News rank rises (Chronicle HE, 25 May 2007; East Valley Tribune, March 18, 2007)
Implications on/for Diversity (1) Increasing vertical stratification w/ growing gap between elite and mass education Public HEIs have hard time competing:...measures favor private institutions over public ones (Chronicle HE, 25 May 2007) Student selectivity indicators and shift in resources being made to improve ranking are disadvantageous for low income and minority students (Clarke, 2007) certain institutions or types of institutions rise to the top regardless of the specific indicators and weightings (Usher and Savino, 2007) As demand for status increases, rankings are leading to creation of more elite institutions. (Samuelson, Newsweek, 2004) Devaluing of hundreds of institutions that do not meet criteria to be included in rankings (Lovett, President AAHE, 2005)
Implications on/for Diversity (2) Despite support for inter-institutional collaboration, in a competitive environment, elite institutions may see little benefit working with/helping lesser institutions. Worldwide comparisons more significant in the future: `Reinforce effects of market-based & competitive forces (Clarke, 2007) Development of single world market Formation of international networks An appropriate public policy role in the development and distribution of league tables is critical.
ellen.hazelkorn@dit.ie