Monitoring University Performance and Adherence to Strategy: Creating a Sustainable Ranking Strategy Dr. Kevin Downing Director Knowledge Enterprise and Analysis City University of Hong Kong
Presentation Outline Part 1: Rising Asia: Sustaining Progress Strategic planning Strategic Implementation Part 2: Integrated Planning for Success Potential outcomes A planning model Part 3: Management Metrics for the Planning Model Departmental Teaching and Learning Indicators Departmental Research and Knowledge Transfer indicators
Three Ranking Systems - Criteria and Weighting (2012) Academic Ranking of World Universities (ARWU) QS World University Rankings (QS-WUR) Times Higher Education of World University Ranking (THE) Per Capita Performance; 10% SCI/ SSCI/ A&HCI Articles; 20% Nobel/ Fields Medal Alumni; 10% Nature/ Science Articles; 20% Nobel/ fields Medal Winner; 20% HiCi Researcher; 20% Int'l Student; 5% Int'l ; 5% Citations Per ; 20% Student Ratio; 20% Academic Peer Review; 40% Employer Review; 10% Industry income innovation; 2,50% Citations research influence; 30% Internationa l outlook staff, students and research; 7,50% Research volume, income and reputation; 30% Teaching the learning environmen t; 30%
QS World University Rankings System Research quality Peer review (40%) Citations per faculty (20%) 60.00% Teaching quality Student faculty ratio (20%) 20.00% Graduate employability Recruiter review (10%) 10.00% International outlook International faculty (5%) International students (5%) 10.00%
Times Higher Education (THE) World University Ranking System Old Criteria and Weighting (2010) New Criteria and Weighting (2011) Research volume, income and reputation, Research Income from Industry, 2.5% Int'l Staff and Student, 5% Teaching & learning 30% environment, 2,50% Teaching 30% the learning environment; 30% Citations, Per Paper, 32.5% Industry income innovation; Citations research influence; 30% International outlook staff, students and research; 7,50% Research volume, income and reputation; 30%
Times Higher Education (THE) World University Ranking System (2010 old version) Criteria Citations research influence Research volume, income and reputation Teaching the learning environment International diversity Industry Income innovation Indicators Citation impact (normalized average citation per paper) (32.5%) Reputational survey (research) (19.5%) Research income (scaled) (5.25%) Papers per research and academic staff (4.5%) Public research income/ total research income (0.75%) Reputational survey (teaching) (15%) PhDs awards per academic (6%) Undergrad. admitted per academic (4.5%) Income per academic (2.25%) PhDs/undergraduate degrees awarded (2.25%) Ratio of international to domestic staff (3%) Ratio of international to domestic students (2%) Research income from industry (per academic staff) (2.5%) Percentage weightings 32.50% 30.00% 30.00% 5.00% 2.50%
Times Higher Education (THE) World University Ranking System (2011 new version) THE Criteria Indicators Reputational survey (teaching) (15%) Percentage weightings Teaching the learning environment Research volume, income and reputation PhDs awards per academic (6%) Undergraduates admitted per academic (4.5%) Income per academic (2.25%) PhDs/undergraduate degrees awarded (2.25%) Reputational survey (research) (18%) Research income (scaled) (6%) Papers per research and academic staff (6%) 30.00% 30.00% Citations research influence International outlook - staff, students and research Industry Income innovation Citation impact (normalized average citation per paper) (30%) 30.00% Ratio of international to domestic staff (2.5%) Ratio of international to domestic students (2.5%) Proportion of internationally co-authored research papers (2.5%) 7.50% Research income from industry (per academic staff) (2.5%) 2.50%
Academic Ranking of World Universities (ARWU) System Criteria Indicators Code Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Percentage weightings Alumni 10% Quality of Staff of an institution winning Nobel Prizes and Fields Medals Highly cited researchers in 21 broad subject categories Award 20% HiCi 20% Papers published in Nature and Science* N&S 20% Research Output Papers indexed in Science Citation expanded and Social Science Citation PUB 20% Per Capita Performance Per capita academic performance of an institution PCP 10% * For institutions specialized in humanities and social sciences such as London School of Economics, N&S is not considered, and the weight of N&S is relocated to other indicators.
Part 1: Rising Asia: Sustaining Progress
Rising Asia: QS-WUR Top 200 (2008 2012) A Strategic Success Story?
Hong Kong Institutions in QS World University Rankings (Top 400) in 2005-2012 http://144.214.193.5/university_rankings/
Strategic Planning (The Easy Part) o Core steering group formation and scheduling o Wide-ranging consultation for planning and process o External environment scanning (various parameters) o Internal environment scan (various parameters) o Plan drafting and further consultation o Publication and dissemination of strategic plan
Some brief Examples of External Scanning (based on Rankings Metrics parameters)
QS WUR- Comparative Analysis of universities in South Korea (Top 200) in 2012 Academic Reputation 100 Seoul National University International Student 80 60 40 20 Employer Reputation KAIST - Korea Advanced Institute of Science & Technology Pohang University of Science And Technology (POSTECH) 0 Yonsei University Korea University International Citation per Sungkyunkwan University Student ratio
QS WUR -Comparative Analysis of universities in Japan (Top 200) in 2012 The University of Tokyo International Students Academic Reputation 100 90 80 70 60 50 40 30 20 10 0 Employer Reputation Kyoto University Osaka University Tokyo Institute of Technology Tohoku University Nagoya University Kyushu University International Citations Per Hokkaido University Waseda University Student Ratio Keio University
QS WUR - Comparative Analysis of universities in China (Top 200) in 2012 Academic Reputation 100 Peking University 90 80 70 Tsinghua University International Students 60 50 40 Employer Reputation Fudan University 30 20 10 Shanghai Jiao Tong University 0 Nanjing University International Citations Per Zhejiang University University of Science and Technology of China Student Ratio
QS WUR - Comparative Analysis of universities in Singapore (Top 200) in 2012 International Students Academic Reputation 100 90 80 70 60 50 40 30 20 10 0 Employer Reputation National University of Singapore (NUS) Nanyang Technological University (NTU) International Citations Per Student Ratio
QS WUR - Comparative Analysis of Universities in Hong Kong (Top 400) in 2012 Academic Reputation 100 80 University of Hong Kong International Students 60 40 Employer Reputation The Hong Kong University of Science and Technology The Chinese University of Hong Kong 20 0 City University of Hong Kong The Hong Kong Polytechnic University Hong Kong Baptist University International Citations Per Student Ratio
QS WUR - Comparative Analysis of Universities in Malaysia (Top 400) in 2012 Academic Reputation 100 80 International Students 60 40 Employer Reputation Universiti Malaya (UM) Universiti Kebangsaan Malaysia (UKM) 20 Universiti Sains Malaysia (USM) 0 Universiti Teknologi Malaysia (UTM) Universiti Putra Malaysia (UPM) International Citations Per Student Ratio
QS WUR - Comparative Analysis of Universities in Israel (Top 400) in 2012 Academic Reputation 100 Hebrew University of Jerusalem 80 International Student 60 40 Employer Reputation Tel Aviv University 20 0 Technion - Israel Institute of Technology Ben Gurion University of The Negev International Citation per Student ratio
QS WUR - Comparative Analysis of Universities in Saudi Arabia (Top 400) in 2012 International Student Academic Reputation 100 90 80 70 60 50 40 30 20 10 0 Employer Reputation King Saud University King Fahd University of Petroleum & Minerals King Abdul Aziz University (KAU) International Citation per Student ratio
QS WUR - Comparative Analysis of Universities in Other Middle East countries (Top 400) in 2012 International Student Academic Reputation 100 90 80 70 60 50 40 30 20 10 0 Employer Reputation American University of Beirut (AUB) United Arab Emirates University International Citation per Student ratio
Part 2: Integrated Planning for Success
Strategic Implementation (The Hard Part) o Publication and dissemination (Easy) o Aligned College/School level plans (Easy) o Goal setting and milestones (Easy) o Achieving adherence to strategic aims (Hard) oan Integrated Planning Model for Success
Hong Kong Institutions in QS World University Rankings (Top 400) in 2005-2012 http://144.214.193.5/university_rankings/
Aligning Metrics for Integrated Planning QA data HR data Research & KT metrics Budget Hearings Financial data Teaching & Learning metrics Qualitative input
Part 3: Management Metrics for the Planning Model
Example Comparative Metrics: Departmental Learning and Teaching Performance
Example Departmental Performance Indicators % International Students Average Entry A-Level Score % to Total Academic Staff % with PhD or Professional Accreditation % Outbound Exchange Students % Graduates with FT Employment (within 6 months of completion) Input Quality Staffing and Resources Output Quality % Self-financed Students Average Entry English Score Number of Students Per % International % Student with Internship Experience
Staircase Model Threshold (One star) Towards Excellence (Two star) Excellence (Three star) Input Quality Staffing and Resources Output Quality
Example Growth Chart (Department X) Current Performance Threshold * (One star) Towards Excellence ** (Two star) Excellence *** (Three star) Transition Delta Average Entry A-Level Score 15.8 0.2 Input Quality Average Entry English Score 3 0.5 % International Students 18% 2% % Self-financed Students 0% 0% Staffing Resources % (A to I Grade) to Total Academic Staff % International (FTE) Number of Studends Per 62% 18% 51% 0% 9-2 % with PhD or Professional Accreditation 91% 9% % Outbound Exchange Students 17% 3% Output Quality % Students with Intership Experience % Graduates with FT Employment (within 6 months of completion) 53% 17% 97.5% 2.5%
Example Growth Chart (Department Y) Current Performance Threshold * (One star) Towards Excellence ** (Two star) Excellence *** (Three star) Transition Delta Average Entry A-Level Score 13.8 0.2 Input Quality Average Entry English Score 1.6 0.9 % International Students 3% 7% % Self-financed Students 39% 1% Staffing Resources % (A to I Grade) to Total Academic Staff % International (FTE) Number of Studends Per 30% 10% 31% 9% 14-1 % with PhD or Professional Accreditation 60% 20% % Outbound Exchange Students 0% 15% Output Quality % Students with Intership Experience % Graduates with FT Employment (within 6 months of completion) 12.5% 17.5% 93% 2%
Example Comparative Metrics: Departmental Research and Knowledge Transfer
Introduction Three indices are used as a working example of potential Performance Indicators (PI s): The Staff Performance The Grant Performance The Postgraduate Supervision Performance The Staff Performance contains 2 PI s: No. of Publications per FTE and No. of Citations per FTE The Grant Performance contains 3 PI s : % of FTE holding External Grants, No. of Grants per FTE and Grant Income per FTE The Postgraduate Supervision Performance contains 2 PI s : No. of PhD students per FTE and % of PhD Completion per FTE * The census period of Publication was 1/7/2009 to 30/6/2010 The census period of Citation was 1/1/2010 to 31/12/2010
Publication Breakdown There are 6 broad categories of output included in Publication Breakdown slides (see below). (Total 27 types of output set with the same weighting for calculating publication performance.) 4 Creative and literary works, consulting reports and case studies 4.1 Authored play, poem, novel, story 4.2 Painting, sculpture, drawing, photograph 4.3 Film, video44 Performance and participation in exhibits 4.5 Translation of other's work 1 Scholarly books, monographs and chapters 1.1 Research book or monograph (author) 4.6 Engineering, architectural, graphic designs 4.7 Computer software or system 1.2 Chapter in an edited book (author) 4.8 Consulting or contract research report 1.3 Textbook (author) 1.4 Edited book (editor) 4.9 Written teaching case study or extensive note 2 Journal publications 2.1 Publication in refereed journal 2.2 Publication in policy or professional journal 5 Patents, agreements, assignments, and companies 5.1 Patents granted 5.2 Licensing agreements 5.3 Assignments of intellectual property rights 3 Conference papers 5.4 Companies 3.1A Invited conference paper (refereed items) 3.1B Invited conference paper (non-refereed items) 6 All other outputs 3.2 Refereed conference paper 6.1 Journal editor 3.3 Other conference paper 6.2 Review of books or of software 6.3 Postgraduate research theses 6.4 Other outputs(other Outputs; Prizes and awards)
Department A:Overall Annual Performance (2009-2010) Staff Performance Grant Performance PhD Supervision Performance # Publications Per FTE (1/7/2009 to 30/6/2010) # Citations per FTE (in 2010 with all papers) % FTEs Holdings an External Grant Current Performance Threshold (A*1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Transition Delta Median 2.33 1 0.02 (to AAA=2.35) 1.68 49.38 1 0 (to Max=49.38) 25.40 15.38% 0.65596905 4.94% (to AAA=20.32%) 14.59% # Grants per FTE 0.50 0.60230271 0.02 (to AAA=0.52) 0.41 Grant Income per FTE 362,556 0.379614325 4,636 (to AAA=367,193) 222,728 # PhD per FTE 1.10 1 0 (to Max=1.1) 0.46 # PhD Completion per FTE (in 2010) 0.33 0.655744033 0 (to Max=0.33) 0.06 Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator
Staff Performance Grant Performance PhD Supervision Performance Department A:Overall Annual Performance (2010-2011) # Publications Per FTE (1/7/2010 to 30/6/2011) # Citations per FTE (in 2011 with all papers) % FTEs Holdings an External Grant Current Performance Threshold (A*1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator Transition Delta Median 2.08 0.664329126 0.01 (to AAA=2.1) 1.63 79.92 1 0 (to Max=79.92) 34.82 38.46% 1 0% (to Max=38.46%) 16.95% # Grants per FTE 0.69 1 0 (to Max=0.69) 0.42 Grant Income per FTE 478,682 1 0 (to Max=478,682) 101,460 # PhD per FTE 1.33 1 0 (to Max=1.33) 0.59 # PhD Completion per FTE (in 2011) 0.27 0.656013211 0 (to AAA=0.27) 0.03
Department A:Overall Outputs Breakdown (2010-2011) Current Performance Threshold (A* 1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Transition Delta Median 1.Scholarly books, monographs and chapters 0.07 1 0 (to AA=0.07) 0.07 2.Journal publications 0.75 0 0 (to Max=0.75) 0.59 3.Conference papers 0.84 0.333333333 0.03 (to AAA=0.87) 0.64 4.Creative and literary works, consulting reports and case studies 5.Patents, agreements, assignments, and companies 0.00 0.623490789 0.16 (to AAA=0.16) 0.00 0.00 1 Max = 0 0.00 6.All other outputs 0.42 0.322536837 0 (to Max=0.42) 0.14 Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator
Department B:Overall Annual Performance (2009-2010) Staff Performance Grant Performance # Publications Per FTE (1/7/2009 to 30/6/2010) # Citations per FTE (in 2010 with all papers) % FTEs Holdings an External Grant Current Performance Threshold (A*1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Transition Delta Median 1.50 0.349233663 0.18 (to AA=1.68) 1.68 27.94 0.054861899 7.79 (to AAA=35.73) 25.40 12.50% 0.62129853 2.09% (to AA=14.59%) 14.59% # Grants per FTE 0.25 0 0.07 (to A=0.32) 0.41 Grant Income per FTE 347,531 0.10428769 19,662 (to AAA=367,193) 222,728 PhD Supervision Performance # PhD per FTE 0.25 0.415315758 0.21 (to AA=0.46) 0.46 # PhD Completion per FTE (in 2010) 0.06 0.233578571 0.15 (to AAA=0.21) 0.06 # Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator
Staff Performance Grant Performance PhD Supervision Performance Department B:Overall Annual Performance (2010-2011) # Publications Per FTE (1/7/2010 to 30/6/2011) # Citations per FTE (in 2011 with all papers) % FTEs Holdings an External Grant Current Performance Threshold (A*1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator Transition Delta Median 1.31 0 0.32 (to AA=1.63) 1.63 29.06 0.061653799 5.76 (to AA=34.82) 34.82 16.67% 0.445619743 0.29% (to AA=16.95%) 16.95% # Grants per FTE 0.39 0.231777232 0.03 (to AA=0.42) 0.42 Grant Income per FTE 143,967 0.31986532 83,679 (to AAA=227,646) 101,460 # PhD per FTE 0.44 0.260729567 0.15 (to AA=0.59) 0.59 # PhD Completion per FTE (in 2011) 0.00 0.117453034 0.03 (to AA=0.03) 0.03
Department B:Overall Outputs Breakdown (2010-2011) Current Performance Threshold (A* 1 ) Towards Excellence (AA* 2 ) Excellence (AAA* 3 ) Transition Delta Median 1.Scholarly books, monographs and chapters 0.07 0.540545837 0.12 (to AAA=0.2) 0.07 2.Journal publications 0.55 0 0.04 (to AA=0.59) 0.59 3.Conference papers 0.46 0.333333333 0.17 (to AA=0.64) 0.64 4.Creative and literary works, consulting reports and case studies 5.Patents, agreements, assignments, and companies 0.00 0.032285217 0.16 (to AAA=0.16) 0.00 0.00 0.275016267 Max = 0 0.00 6.All other outputs 0.22 0.338095201 0.05 (to AAA=0.27) 0.14 Remarks: *1 - A stands for 25 percentile to median of College of Business of that indicator *2 - AA stands for median to 75 percentile of College of Business of that indicator *3 - AAA stands for 75 percentile to maximum of College of Business of that indicator
Aligning Metrics for Integrated Planning QA data HR data Research & KT metrics Budget Hearings Financial data Teaching & Learning metrics Qualitative input
Pulling it All Together Needs: a) Ensure adherence to strategic goals b) Recognise different disciplines c) Motivate staff d) Reward departmental progress e) Develop competitive awareness Methods: a) Align indicators to institutional strategic goals b) Use College/ means for internal benchmarks c) Align with individual PBPR d) Evidence-based budget allocation (Hearings) e) Longitudinal and crosssectional comparisons
Thank you!