A Comparison of Academic Ranking Scales

A Comparison of Academic Ranking Scales Alona Zharova Andrija Mihoci Wolfgang Karl Härdle Ladislaus von Bortkiewicz Chair of Statistics C.A.S.E. Center for Applied Statistics and Economics Collaborative Research Center 649: Economic Risk Humboldt Universität zu Berlin lvb.wiwi.hu-berlin.de case.hu-berlin.de sfb649.wiwi.hu-berlin.de

Motivation - Nature

Motivation -2 Nobel Prize in Economics Figure : Selected Nobel Prize winners (from 75), 969-24 (link) Milton Friedman on BBI:

Motivation -3 Nobel Prize in Economics Figure 2: Selected Nobel Prize winners (from 75), 969-24 Robert C. Merton on BBI:

Motivation -4 Rankings and Funds Asset Allocation at Universities Distribution scheme based on performance, e.g., HB ranking Recruitment process for sta Funding Organisations Federal Ministry of Education and Research in Germany Deutsche Forschungsgemeinschaft (DFG): mechanisms, CRCs

Motivation -5 Objectives (i) Treatment of incomplete ranking data Statistics and Scientometrics Handelsblatt (HB), RePEc (RP) and Google Scholar (GS) (ii) Ranking systems dependence Perspectives: ranks and dynamics Case study: age groups across ranks

Motivation -6 Example Research project leaders Collaborative Research Center 649: Economic Risk - 35 research projects, 38 principal investigators (PIs), 25 25 Lifespan: I. (25 28); II. (29 22); III. (23 26) Comparison using Handelsblatt (HB), RePEc (RP) and Google Scholar (GS) ranking data

Motivation -7 Research Questions How to predict an academic rank for researchers which are not included in a particular ranking system? How are HB, RP and GS rankings related? How does age inuence academic ranking: ranking scores, dynamics, number of citations?

Outline. Motivation 2. Academic Rankings and Incomplete Data 3. Ranking Systems Dependence 4. Empirical Results 5. Conclusions

Academic Rankings and Incomplete Data 2- Academic Rankings Scientometrics - measuring of scientic performance Academic ranking systems HB 6 rankings VWL/BWL: Life's Work (LW), Current Researchers (CR) and Under 4 (U4), Wohlrabe (23) here: 726 researchers RP 37 rankings, here: 233 researchers GS proles

Academic Rankings and Incomplete Data 2-2 Data VWL BWL LW CR U4 LW CR U4 PI 5.54 3.43 - - - - PI2 6.9 - - - - - PI3 8.78 3.8 - - - - Table : Ranking scores of selected project's PI of CRC 649 within HB VWL (23) and BWL (24) LW, CR and U4 rankings

Academic Rankings and Incomplete Data 2-3 Data HB RP GS PI 5 276 2953 PI2 8 695 - PI3 9-3744 PI4 - - 642 Table 2: Ranking scores of selected project's PI of CRC 649 within HB (VWL LW), RP (total score) and GS (citations)

Academic Rankings and Incomplete Data 2-4 Data: Merging No HB Yes RP Yes No 233 24 Figure 3: Mosaic plot of number of researchers, when merging of HB, RP and GS rankings takes place (Yes/No). The number of GS proles is here gurative. 726 328 2 7 YesNo Yes No GS

BWL VWL Ranking Systems Dependence 3- HB 2 23 LW 2 2 23 CR 2 2 23 U4 Figure 4: Parallel coordinates plots for scores of researchers within HB VWL from 2 to 23. 29 22 24 LW 29 22 24 CR 29 22 24 U4

BWL VWL Ranking Systems Dependence 3-2 HB 2 23 LW 2 2 23 CR 2 2 23 U4 29 22 24 LW 29 22 24 CR 29 22 24 U4 Figure 5: Parallel coordinates plots for scores of researchers within HB BWL from 29 to 24.

Q-Q plot Scatter plot Ranking Systems Dependence 3-3 Regression 3 3 3 3 3 5 5 5 5 5 5 5 5 5 2 4 4 5 5 5 5 2-5 5 VWL CR -5 5 VWL U4-5 5 BWL CR -5 5 BWL U4-5 5 BWL LW Figure 6: Scatter plot and QQ plot of the relationship of HB VWL LW ranking with other ve rankings for top - researchers

Q-Q plot Scatter plot Ranking Systems Dependence 3-4 Regression 3 3 3 3 3 5 5 5 5 5 5 5 5 5 2 4 4 5 5 5 5 2-2 2 VWL CR -2 2 VWL U4-2 2 BWL CR -2 2 BWL U4-2 2 BWL LW Figure 7: Scatter plot and QQ plot of the relationship of HB VWL LW ranking with other ve rankings for top - researchers

Q-Q plot Scatter plot Ranking Systems Dependence 3-5 Regression 3 3 3 3 3 5 5 5 5 5 5 5 5 5 2 4 5 5 2 5 5-5 5 VWL CR -5 5 VWL U4-5 5 BWL CR -5 5 BWL U4-5 5 BWL LW Figure 8: Scatter plot and QQ plot of the relationship of HB VWL LW ranking with other ve rankings for top - researchers

Ranking Systems Dependence 3-6 Regression Top Top - Top - β β β β β β VWL CR -.25 3.69-2.8 3.83 -.43 3.76 VWL U4.65 3.68 4.5 3.3.9 3.54 BWL CR -3.7 3. -8.4 3.78-2.69 3. BWL U4.7 2.77 -.22 2.98.53 2.62 BWL LW.3.99 7.2.65-2.3.24 Table 3: Estimates of the regression model parameters for rankings between HB VWL LW (dependent variable) and other ve rankings (explanatory variables) for researchers in 23-24.

GS Ranking Systems Dependence 3-7 RP and GS # 5 2 2 3 RP # 4 Figure 9: Scatterplot of RP and GS citations for 24 researchers in 25; correlation coecient is.7

Citations Ranking Systems Dependence 3-8 GS 27 28 29 2 2 22 23 24 Figure : Parallel coordinates plot for GS citations from 27 to 24 for researchers. Max = 2953 citations

Ranking scores Ranking Systems Dependence 3-9 HB, RP and GS HB RP GS Figure : Parallel coordinates plot for three variables (HB common score, RP total score and GS citations) on 7 researchers

GS RP RP Ranking Systems Dependence 3- HB, RP and GS # 4 6 4 2 3 2 3 2 5 3 HB 3 6 GS # 4 5 3 HB Figure 2: Scatterplots of HB common score, RP total score and GS citations for 7 researchers

HB RP GS Empirical Results 4- Age 3 5 4 # 5 2 3 5 7 9 8 3 5 7 9 Age 3 5 7 9 Figure 3: Scatterplots for age and ranking scores of HB, RP and GS for Top-7 scientists within each ranking system

Empirical Results 4-2 Age groups Age Ranking RP HB GS <35 36 4 4 45 46 5 5 55 56 6 6 65 66 7 75> Figure 4: Mosaic plot of HB, RP and GS for Top-7 scientists within each ranking system. The width of the columns represents the number of persons within each age group

Conclusions 5- Conclusions (i) Treatment of incomplete ranking data Statistics and Scientometrics Handelsblatt (HB), RePEc (RP) and Google Scholar (GS) (ii) Ranking systems dependence Perspectives: ranks, dynamics Case study: age groups across ranks

References 6- References Wohlrabe, K. Einige Anmerkungen zum Handelsblatt-Ranking 23 ifo Schnelldienst, 79-83, 23. Iselin, D., Schläpfer, J. and Vogel S. Die publikationsstärksten Volkswirte: das Handelsblatt-VWL- Ranking 23 www.oekonomenstimme.org (accessed 3 Nov 24), 23. Zimmermann, C. Academic Rankings with RePEc Econometrics, MDPI, Open Access Journal, (3): 249-28, 23.