Lessons learnt case study (II)

Lessons learnt case study (II) Developing a Composite Indicator of Scientific and Technological Research Excellence 12 th JRC Annual Training on Composite Indicators & Multicriteria Decision Analysis (COIN 2014) daniel.vertesy@jrc.ec.europa.eu European Commission Joint Research Centre Econometrics and Applied Statistics Unit Composite Indicators Research Group (JRC-COIN) Lessons Learnt Case Study II: Reserach Excellence 1 Lessons Learnt Case Study II: Reserach Excellence 2 OVERVIEW [STEP #] Background and Motivation Conceptual framework [1] Defining and Measuring Research Excellence [1-4] Normalization, Weighting and Aggregation [5-6] Uncertainty analysis [7] Validation [8] Section 1 BACKGROUND AND MOTIVATION * Work carried out with Sjoerd Hardeman, Vincent van Roy, Michaela Saisana Lessons Learnt Case Study II: Reserach Excellence 3 Lessons Learnt Case Study II: Reserach Excellence 4

Background and Motivation The broader context Why assess national research systems? Descriptive account: research in contemporary society Normative account: policy interest in accountability and competitiveness Explanatory account: the new economics of research Why focus on excellence? Political interest by European Commission (DG RTD) JRC Research project aimed at exploring national research systems: What characterizes national research systems and how and to what extent do countries differ? Mapping country performance through [Composite] Indicators What determines the efficiency of national research systems? Applying a non-parametric production frontier analysis Create Composite Indicators only when appropriate! Lessons Learnt Case Study II: Reserach Excellence 5 Lessons Learnt Case Study II: Reserach Excellence 6 History of iterations Section 2 REFINING THE CONCEPTUAL FRAMEWORK Feasibility Study (Barré et al., 2011): Conclusion: Out of the 22 indicators deemed feasible to collect by the Barré et al Report, 6+4+3 could statistically fit an alternative framework to measure public and business research excellence and collaborations Presentation of Initial aggregation methodology, Expert Workshop (Ispra, Oct 2012) Conclusions: More focused conceptual and theoretical foundation; (i.e., Research innovation; Research inputs research outputs; Research interactions research outputs) Presentation of updated framework and results; Discussions with policy users (2013) Conclusions: further fine-tune some variables; issue of time scope; geographical coverage Final Report (Sep 2013) Lessons Learnt Case Study II: Reserach Excellence 7 Lessons Learnt Case Study II: Reserach Excellence 8

National Research Systems /1 National Research Systems /2 Components Research assets (actors) Structural capabilities (sectorial and disciplinary composition) Relationships Research interactions Attributes Different type of actors: universities, industry, government Different dimensions to interactions: geographical, cognitive, and institutional Goal orientation Research excellence Lessons Learnt Case Study II: Reserach Excellence 9 Lessons Learnt Case Study II: Reserach Excellence 10 Definition of Research Excellence The better the definition = the better the measurement knowledge producing activities whose outputs are considered of high-end quality (Tijssen, 2003) 1. Output oriented 2. focus on the quality rather than the quantity 3. focus on top-end rather than average Section 3 DEFINING AND MEASURING RESEARCH EXCELLENCE Lessons Learnt Case Study II: Reserach Excellence 11 Lessons Learnt Case Study II: Reserach Excellence 12

Defining Research & Excellence Issues of Measurement /1 [S&T] Research (OECD, 2002) = creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications Research Excellence (our definition, based on Tijssen, 2003) = the high quality outcome of research 1. Output oriented 2. focus on the quality rather than the quantity 3. focus on top-end rather than average Science and Technology: different ways of evaluation (peers vs. market) Main quality issues of indicators: Extent: scale and scope of the data; type of activities covered How many countries? 27 EU member states + comparator countries At least two time points Reliability: systematic collection of data, consistency over time; i.e., Did definitions change over time? Validity: how well does the data correspond to the phenomenon to be measured? Measures research (as defined); a measure of output; focus on quality rather than quantity; not average but top-end quality Lessons Learnt Case Study II: Reserach Excellence 13 Lessons Learnt Case Study II: Reserach Excellence 14 15 16 Issues of Measurement /2 Does the data represent research? IGNORE DATA Some examples Number of publications per public sector researchers? Does the data represent high-end quality aspects of research? Does the data represent research outputs? Does the data represent research inputs (or process) whose outputs are not yet covered by a strong indicator? Specialization in publications in the fields of the Grand Societal Challenges? Share of foreigners in doctoral programmes? => IGNORE these indicators INCLUDE AS STRONG INDICATOR INCLUDE AS ADDITIONAL WEAK INDICATOR Lessons Learnt Case Study II: Reserach Excellence 15 Lessons Learnt Case Study II: Reserach Excellence 16

Indicators of research excellence /1 Indicators of research excellence /2 The top-end quality output of scientific research: Excellence of main actors of the research system: Highly cited publications (i.e., top 10%) [HICIT] Extent (+), reliability (+), and validity (+) Highly cited researchers Extent (-), reliability (+), and validity (+) (data not available) Nobel Prizes & Fields Medals Extent (-), reliability (-), and validity (+) Top 500 research organizations (Leiden ranking) Extent (+), reliability (+), and validity (+/-) ERC research grants Extent (+/-), reliability (+), and validity (+/-) Lessons Learnt Case Study II: Reserach Excellence 17 Lessons Learnt Case Study II: Reserach Excellence 18 Indicators of research excellence /3 Never underestimate the choices to make! High-quality technological outcomes of research activities: PCT patent applications Extent (+), reliability (+), and validity (+/-) Triadic patent families (TPF) Extent (-), reliability (+), and validity (+/-) Patent quality indicators (OECD) Extent (-), reliability (+), and validity (+/-) Contractual research Extent (-), reliability (-), and validity (-) PCT/mln pop 400 350 300 250 200 150 50 0 JP-08 JP-10 JP-09 JP-07 JP-05 JP-06 JP-04 JP-03 JP-02 JP-01 JP-00 0 50 150 TPF/mln pop Decide on the relevant indicators Consider alternative ways to measure, collect relevant data Consider alternative specification i.e., meaning of different numerators / denominators Consider implications meaning, statistical behaviour Long iterative process!!! Lessons Learnt Case Study II: Reserach Excellence 19 Lessons Learnt Case Study II: Reserach Excellence 20

How to Best Specify the Indicators? Scale-normalization An example: Universities and Public research organizations (PRO) Aim: simple, possibly uniform solution; high correlation To decide: Source = Leiden ranking vs. ARWU, etc.; SCIMAGO vs. Content = Universities + PRO, how to harmonize? Numerator: Which measure of excellence? How to aggregate from institution score to country scores? What threshold to choose? Denominator: Staff, Population, R&D, etc.? Correlation between Research Excellence indicators and potential normalizing factors Indicator Country group GDP Population GERD Other: HICIT EU-27 0.989 0.987 0.981 Total 0.997 All Countries 0.942 0.942 0.461 Publications 0.972 TOPINST EU-27 0.686 0.587 0.684 0.751 All Countries 0.880 0.184 0.901 0.791 PCT EU-27 All Countries 0.982 0.919 0.996 0.959 0.973 0.294 Number of Researchers 0.984 0.817 ERC EU-27 0.887 0.799 0.832 0.886 All Countries 0.984 0.978 0.972 0.988 Lessons Learnt Case Study II: Reserach Excellence 21 Lessons Learnt Case Study II: Reserach Excellence 22 Indicators selected Code Name Definition Sources HICIT Highly cited Field-normalised count of the 10 % most highly Science Metrix publications per total cited publications divided by total publications (Scopus) publications with an author from that country (years reliable data available: 2000-2007) PCTPAT PCT Patent Patent applications filed under PCT by inventors OECD applications per country of residence (fractional counting) in all million inhabitants IPC classes per million inhabitants (years reliable data available: 2000-2008) TOPINST Top universities and Number of top 250 world scientific universities SciMago Institute public research and top 50 public research organisations in a Ranking (Scopus) institutes per total country divided by total R & D expenditures R & D expenditure (years available: 2003-2007, 2004-2008) ERC ERC grants received per public R & D expenditure Value of ERC grants received by country of host organisation, equally spread over project duration divided by public R & D expenditures (years available: 2007-2011) ERC, CORDIS DG-RTD Issues of time coverage Aimed for coherence (in order to be able to test other indicators) Constraints: Low overlap Lags due to citation-window for highly cited publications Demand for up-to-date indicators Years Indicator 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 HICIT x x x x x x x x x TOPINST????? PCT (x) (x) (x) (x) (x) (x) x x x x x ERC (x) x x x 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 HICIT x x x x x x x x x TOPINST????? PCT (x) (x) (x) (x) (x) (x) x x x x x Solution chosen: 2 time points selected: 2005 and 2008 for 3-indicators framework covering 41 countries; and 2007 and 2008 for 4 indicator framework covering ERA countries; but labeled as 2005 and 2010 to bear in mind the lag. Lessons Learnt Case Study II: Reserach Excellence 23 Lessons Learnt Case Study II: Reserach Excellence 24

0 0 0 Description of Data Section 4 NORMALIZATION, WEIGHTING AND AGGREGATION Indicators: Years considered: HICIT (Number of highly cited Publications per total publications) TOPINST (No. of Top 250 universities and Top 50 research institutes per GERD) PCTPAT (Number of PCT Patent Applications per mln. population) ERC (Value of ERC project grants, per public R & D expenditure) 2003 & 2007 2008 & 2010 2004 & 2008 2008 & 2010 Nr. of Observations 82 82 82 68 Missing a ( %) 0 % 0 % 0 % 17 % Min 3.8 0.0 0.8 0.0 Max 18.2 927.0 348.9 14.3 Mean 10.1 166.1 82.4 2.1 Standard Deviation 3.9 207.4 92.8 3.0 Skewness 0.1 1.7 1.1 2.3 Kurtosis -1.1 2.9 0.2 6.0 Country-year combination Outliers to be treated! Lessons Learnt Case Study II: Reserach Excellence 25 Lessons Learnt Case Study II: Reserach Excellence 26 Variable scores / 1 Variable scores / 2 HICIT ( 2010 ).02.04.06.08 0.1 TOPINST ( 2010 ).001.002.003.004 5 10 15 20 HICIT 0 200 400 600 800 0 TOPINST PCTPAT ( 2010 ).015.005.01 ERC ( 2010 ).1.2.3.4 0 200 300 400 PCTPAT 0 5 10 15 ERC Lessons Learnt Case Study II: Reserach Excellence 27 Outliers to be treated! Lessons Learnt Case Study II: Reserach Excellence 28

29 30 Log-transforming ERC Correlation Outliers treated Descriptives N Missing (%) Min Max Mean StDev skewness kurtosis skew>2& kurt>3.5 N. Outliers ERC LnERC 68 82 17% 17% 0.0-3.3 14.3 2.7 2.1-1.0 3.0 2.1 2.3 0.0 6.0-1.6 1 0 3 0 ERC - LOG-TRANSFORMED 3.0 2.0 1.0 0.0 0.0 5.0 10.0 15.0 20.0-1.0-2.0-3.0 HICIT TOPINST PCTPAT ERC a HICIT 1 *** 0.674 *** 0.745 *** 0.6320 *** TOPINST 0.674 *** 1 *** 0.758 *** 0.526 *** PCTPAT 0.745 *** 0.758 *** 1 *** 0.465 *** ERC a 0.632 *** 0.526 *** 0.465 *** 1 *** NB: *** = significant at 1 %; 40 countries (and EU27 weighted average) at 2 time points combined; a) correlation scores after outlier treatment Cluster analysis Cluster 1 n = 5 countries Cluster 2 n = 14 countries List of countries CH, DK, IL, NL, SE AT, BE, CZ, DE, FI, FR, GR, IT, JP, KR, NO, SK, UK, US Cluster 3 n = 21 countries BG, BR, CN, CY, EE, ES, HR, HU, IE, IN, IS, LT, LU, LV, MT, PL, PT, RO, RU, SI, TR Entire dataset n = 40 countries -4.0 ERC List of indicators Benchmark country with highest score per variable Highly cited publications CH BE IS CH PCT patents SE FI IE SE Top institutes CH BE BG CH Lessons Learnt Case Study II: Reserach Excellence 29 Lessons Learnt Case Study II: Reserach Excellence 30 31 Aggregation Arithmetic vs. geometric? Geometric reflects excellence = countries need to be good in all aspects Section 5 UNCERTAINTY ANALYSIS Lessons Learnt Case Study II: Reserach Excellence 31 Lessons Learnt Case Study II: Reserach Excellence 32

Uncertainty Analysis Steps and considerations 33 1. Uncertainty in the weights for the variables Reference Alternative HICIT 25 % [15 %-35 %] TOPINST 25 % [15 %-35 %] PCTPAT 25 % [15 %-35 %] ERC 25 % [15 %-35 %] 2. Uncertainty in the aggregation formula Reference Alternative geometric average arithmetic average Lessons Learnt Case Study II: Reserach Excellence 33 Uncertainty Results RE Index "2005" RE Index "2010" Rank Interval Rank Interval Switzerland 1 [1, 1] 1 [1, 1] Netherlands 2 [3, 3] 2 [2, 4] Israel 3 [2, 2] 5 [2, 3] Denmark 4 [4, 5] 3 [4, 5] Sweden 5 [4, 5] 4 [3, 5] Finland 6 [6, 8] 6 [6, 8] Germany 7 [7, 10] 7 [7, 9] Belgium 8 [6, 9] 8 [6, 8] United Kingdom 9 [7, 9] 9 [7, 9] EU-27 10 [10, 14] 13 [10, 16] France 11 [11, 15] 12 [12, 15] Austria 12 [11, 13] 11 [10, 13] Italy 13 [12, 18] 14 [14, 17] Greece 14 [14, 18] 18 [16, 19] Spain 15 [15, 20] 17 [15, 19] Norway 16 [11, 19] 10 [10, 13] Ireland 17 [17, 20] 16 [13, 19] Hungary 18 [14, 20] 19 [16, 23] Cyprus 19 [7, 21] 21 [12, 25] Iceland 20 [10, 24] 15 [11, 20] Czech Republic 21 [18, 25] 20 [18, 26] Slovenia 22 [20, 24] 22 [21, 26] Portugal 23 [22, 26] 23 [21, 26] Bulgaria 24 [21, 27] 25 [21, 28] Luxembourg 25 [21, 27] 27 [20, 28] Poland 26 [26, 30] 26 [26, 31] Estonia 27 [24, 28] 24 [23, 27] Slovakia 28 [24, 29] 29 [22, 28] Malta 29 [26, 29] 30 [20, 29] Romania 30 [29, 34] 28 [30, 33] Lithuania 31 [30, 32] 31 [29, 31] Turkey 32 [30, 34] 32 [30, 32] Latvia 33 [30, 33] 34 [34, 34] Croatia 34 [31, 34] 33 [32, 33] 34 Lessons Learnt Case Study II: Reserach Excellence 34 36 Comparing Research Excellence Scores with indicators of Innovation and Competitiveness CH CH CH Section 6 VALIDATION 20 40 60 80 BG RO TR LT LV PL SK HR GR HU MT ES IT CZ PT NO CY EE SI EU27 FR IE NL AT IS LU BE UK FI DE DK SE 20 40 60 80 TR GR RO BG PL SK HR IT LT ES HU CY CZ PT SI LV AT EU27 FR BE IL IS EE DE NO IE LU MT DK NL FI UK SE 20 40 60 80 GR ROSK HR HU BG CY SIPT IT PL MT LTR LV CZ ES EE EU27 IS IE IL FR LU BE DK AT NO NL SE DEFI UK.2.4.6.8 IUS Summary Innovation Index, 2010 IUS Source: European Commission, IUS 2011; Corr = 0.860 30 40 50 60 70 INSEAD Global Innovation Index 2012 GII Source: INSEAD, 2012; Corr = 0.748 4 4.5 5 5.5 6 WEFORUM Global Competitiveness Index 2012 GCI Source: World Economic Forum, 2012; Corr = 0.838 Lessons Learnt Case Study II: Reserach Excellence 35 Lessons Learnt Case Study II: Reserach Excellence 36

37 38 Further exploring correlations Correlations, no causal relations! 20 40 60 80 LU GR PT LV IT PL 20 40 60 80 DE ES HU DK NO CH IS BE FR SE AT IE EU27 GR ES HU BG CY CZ PT SI EE PL SK RO MT TR LT LVHR UK CZ IL FI IT SE FR DK FI DE BE UK AT IS CH NL IE 0 20000 40000 60000 80000 GDP/capita, USD PPP k=2005, 2008 GDP: World Bank WDI; Corr.=0.448 NO LU Interpretation, cautionary remarks, recommendations Use the proposed composite indicator as an input to the broader debate on measuring and monitoring research activities at the country level; a necessary but also preliminary step to inform research policymakers. results conditional upon the conceptual choices made (and sometimes were forced to make due to our ignorance) and the data that we used (and sometimes were restricted in using due to a lack in available alternatives) Next steps: necessity of collecting and using alternative data and methods for the analysis Policy recommendations: Focus more on establishing research excellence embodied by people Improve either scientific or technological research excellence.8.85.9.95 PISA Overall average scores, 2000 PISA scores source: OECD; Corr = 0.532 Lessons Learnt Case Study II: Reserach Excellence 37 Lessons Learnt Case Study II: Reserach Excellence 38