Composite indicators Andrea Saltelli andrea.saltelli@jrc.it OECD WORKSHOP ON INDICATORS OF REGULATORY MANAGEMENT SYSTEMS EXPERT MEETING 2-3 April 2009 BERR CONFERENCE CENTRE LONDON, UNITED KINGDOM 1
Course based on: Joint OECD-JRC handbook. 5 years of preparation, 2 rounds of consultation with OECD high level statistical committee, finally endorsed March 2008 with one abstention 2
How many composites are around? Searching composite indicators on Scholar Google: October 2005 992 Scholar Google June 2006 1,440 May 2007 1,900 October 2008 3,030 February 2009 3,300 3
4 Analytic work based on CI s
See http://www.oecd.org/dataoecd/20/52/38859413.pdf From Jean Philippe Cotis and Romain Duval, Competitiveness, Innovation and economic growth Istanbul 2007, Conference Measuring and Fostering Progress 5
See www.education-economics.org From Ludger Wößmann Contribution of Education and Training to Innovation and Growth Symposium on the Future Perspectives of European Education and Training for Growth, Jobs and Social Cohesion, Brussels, 2007 6
7 Composite indicators and the media
Our rankings Business Economics Big Mac index: A light-hearted guide to whether currencies are at their correct level All Economics rankings» Education Living Politics Global peace index: The world's most and least peaceful countries All Politics rankings» Risk Technology Use: Rankings from the Economist web edition, March 30, 2009 8
9 Economically literate press does have appetite for statistic based narratives.
Jean Pisani-Ferry and Andre Sapir argument on league tables 10
[ ] civil societies learn from the experience of others. Such policy learning can be enhanced by initiatives that facilitate cross country comparison and benchmarking. A telling example in this respect is [ ] PISA. 11
Transparency benefits the democratic process as it empowers national electorates to review the performance of their own governments and it helps focus the debate on key areas of underperformance. The use of league tables facilitates this process. 12
Tito Boeri and the World Bank s Doing Business report, speaking at the Italian Rai3, March 16 2009 13
A ten-pillars snapshot of doing business in different countries 181 economies, from Afghanistan to Zimbabwe 14
15
Ranks for Italy 65 53 83 75 58 84 53 128 60 156 27 Italy lost six position in a year, says Boeri, then he goes into the details. Paying taxes and enforcing contracts 16
International university rankings: How reliable are these rankings? THES World University Rankings What can we do to improve our position on the international scene? Jiao Tong ranking of World Universities 17
18 Can good practices help? Ten steps to build Composite Indicators
Ideally, a composite indicator should be based on: -a solid theoretical framework, -underlying data of good quality, -a defensible construct. 19
A good technical preparation for a CI can make it more robust (to uncertainties in data, weights, ) more resilient (remain relevant over time), more defensible (in dialogue with stakeholders ) 20
Step 1. Developing a solid theoretical framework What is badly defined is likely to be badly measured The challenges are: To integrate a broad set of (probably conflicting) points of view while keeping within a manageable construct To have a community of peers (individuals, regions, countries) willing to accept the theoretical framework. 21 Examples
Examples Doing Business provides a quantitative measure of regulations for starting a business, dealing with construction permits, employing workers, registering property, getting credit, protecting investors, paying taxes, trading across borders, enforcing contracts and closing a business [ ]. 22
Examples Doing Business does not measure all aspects of the business environment [ ]. It does not, for example, measure security, macroeconomic stability, corruption, the labor skills of the population, the underlying strength of institutions or the quality of infrastructure [ ]. 23
The Canadian Council on Learning has developed the Composite Learning Index (http://www.ccl-cca.ca/). The framework: Learning to Know Learning to Do Learning to Live Together Learning to Be[*] Jacques Delors **+ Pillars from Jacques Delors Task Force: UNESCO's International Commission on Education for the Twenty-first Century. 24
I. Learning to Know involves developing the foundation of skills and knowledge needed to function in the world. This includes literacy, numeracy, general knowledge and critical thinking. II. Learning to Do refers to the acquisition of applied skills. It can encompass technical and hands-on skills and knowledge, and is closely tied to occupational success. 25
III. Learning to Live Together involves developing values of respect and concern for others, fostering social and inter-personal skills, and an appreciation of the diversity of Canadians. This area of learning contributes to a cohesive society. IV. Learning to Be refers to the learning that helps develop the whole person mind, body and spirit. This aspect concerns personal discovery, self-knowledge, creativity and achieving a healthy balance in life (~Maslow s top). 26
Step 1. Developing a solid theoretical framework After Step 1. the developer should have A clear understanding and definition of the multidimensional phenomenon to be measured. A nested structure of the various sub-groups of the phenomenon. A list of selection criteria for the underlying indicators, e.g., input, process, output. 27
Step 2. Selecting indicators A composite indicator is above all the sum of its parts Excerpt: The strength of a composite indicator can largely depend on the quality of the underlying data. * +. The theoretical framework should guide the choice of the underlying indicators. The selection process can be quite subjective and therefore should involve stakeholders. Moreover, depending on availability of data, certain indicators cannot be used and proxies need to be considered. 28
Eigenvalues Step 3. Multivariate analysis Analysing the underlying structure of the data is an art a1 1 a2 0.85 1 a3 0.83 0.87 1 a4 0.77 0.81 0.89 1 a5 0.56 0.55 0.72 0.72 1 a6 0.64 0.61 0.66 0.73 0.56 1 b1 0.50 0.72 0.69 0.63 0.40 0.54 1 b2 0.61 0.80 0.75 0.74 0.62 0.54 0.86 1 b3 0.43 0.39 0.54 0.54 0.39 0.48 0.10 0.29 1 b4 0.35 0.26 0.41 0.30 0.31 0.37 0.16 0.24 0.75 1 b5 0.70 0.49 0.48 0.45 0.60 0.31 0.09 0.38 0.21 0.13 1 b6 0.55 0.75 0.78 0.77 0.59 0.57 0.77 0.83 0.44 0.24 0.29 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 8 7 6 5 4 3 2 1 0 0 2 4 6 8 10 12 14 Component number 29 Component Initial Eigenvalues Total % of variance Cumulative % 1 7.242 60.35 60.35 2 1.523 12.69 73.04 3 1.178 9.82 82.86 4 0.554 4.61 87.47 5 0.512 4.26 91.73 6 0.385 3.20 94.94 7 0.242 2.01 96.95 8 0.131 1.09 98.04 9 0.098 0.82 98.04 10 0.064 0.54 99.40 11 0.043 0.36 99.76 12 0.029 0.24 100.00
Step 4. Imputation of missing data. The idea of imputation could be both seductive and dangerous Almost all datasets contain missing data. Country a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6 AT 0.97 0.78 0.84 0.53 0.72 0.34 0.42 0.18 0.49 0.19 0.85 0.02 BE 0.97 0.72 0.82 0.59 0.86 0.45 0.43 0.18 0.49 0.18 0.87 0.02 BG 0.75 0.31 0.54 0.20 0.61 0.16 0.03 0.01 0.17 0.05 0.44 0.01 CY 0.88 0.47 0.67 0.42 0.69 0.20 0.12 0.07 0.44 0.06 0.54 0.01 CZ 0.95 0.71 0.77 0.40 0.77 0.31 0.22 0.09 0.31 0.08 0.87 0.01 DE 0.95 0.78 0.90 0.61 0.80 0.47 0.52 0.24 0.52 0.19 0.76 0.05 DK 0.97 0.84 0.93 0.80 0.39 0.36 0.33 0.62 0.21 0.93 0.04 EE 0.94 0.62 0.74 0.39 0.78 0.26 0.13 0.07 0.24 0.11 0.93 0.03 EL 0.93 0.60 0.52 0.37 0.72 0.35 0.08 0.06 0.71 0.01 ES 0.94 0.49 0.76 0.49 0.90 0.27 0.16 0.08 0.30 0.11 0.81 0.02 EU27 0.93 0.63 0.80 0.49 0.77 0.34 0.29 0.15 0.41 0.14 0.77 0.03 FI 0.99 0.81 0.97 0.70 0.91 0.47 0.19 0.15 0.53 0.15 0.91 0.03 FR 0.96 0.57 0.82 0.52 0.89 0.35 0.43 0.11 0.76 0.02 30
Three common approaches to deal with missing data: case deletion (removes either country or indicator from the analysis) single imputation (e.g. Mean/Median substitution, Regression, etc.) multiple imputation (e.g. Markov Chain Monte Carlo algorithms). Trying to minimize bias and to keep expensive to collect data that would otherwise be discarded. 31
Step 5. Normalisation of data Avoid adding up apples and oranges Ranking Standardization Re-scaling Distance to reference country Categorical scales Cyclical indicators Balance of opinions 32
Step 6. Weighting and aggregation The relative importance of the indicators can become the substance of a negotiation Weights based on statistical models Principal component/factor analysis Data envelopment analysis Regression approach Unobserved components models 33
Step 6. Weighting and aggregation The relative importance of the indicators can become the substance of a negotiation Weights based on opinions: participatory methods Budget allocation Public opinion Analytic hierarchy process Conjoint analysis 34
Step 7. Robustness and sensitivity Uncertainty analysis can be used to assess the robustness of composite indicators Space of alternatives Performance index 60 Weights Imputation 50 40 Aggregation... 30 Including/ Normalisation excluding variables 20 10 Spain Italy Greece 35
Economic and Social Well-being Step 8. Links to other variables Composite indicators can be linked to other variables and measures Comparing effectively complex dimensions: Figure 1. Relationship between the Composite Learning Index and the Economic and Social Well-Being Index in Canada. 100 90 80 70 60 50 y = 0.7691x + 20.249 R 2 = 0.6979 40 40 50 60 70 80 90 100 Composite Learning Index 36
Step 9. Back to the details De-constructing composite indicators can help extend the analysis Ad hoc clustering 37
Step 10. Presentation and dissemination A well-designed graph can speak louder than words The four-quadrant model of the Composite Learning Index (Canadian Council on Learning) 38
39 In house CI development
Concept: (WHO report) Published in PLoSMedicine Sensitivity analysis Results Policy message 40 The Alcohol Policy Index (New York Medical College)
Economic and Social Well-being CCL Report Framework Sensitivity analysis Scenario Pillar Structure Normalisation Weighting Aggregation CLI Preserved z-scores FA within pillar, Regression Linear weights to Factors, FA pillars, Regression weights to pillars S1 Preserved z-scores FA within pillar, FA pillars Linear S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric S12 Preserved Min-max FA within pillar, FA pillars Geometric S13 Not preserved z-scores FA all indicators Geometric S14 Not preserved Min-max FA all indicators Geometric S15 Preserved z-scores FA within pillar, EW pillars Geometric S16 Preserved Min-max FA within pillar, EW pillars Geometric S17 Not preserved z-scores EW all indicators Geometric S18 Not preserved Min-max EW all indicators Geometric S19 Preserved z-scores EW within pillar, EW pillars Geometric S20 Preserved Min-max EW within pillar, EW pillars Geometric S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria S22 Not preserved Raw data FA all indicators Multi-criteria S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria S24 Not preserved Raw data EW all indicators Multi-criteria S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria 41 Results The Composite Learning Index (Canadian Council on Learning) Figure 1. Relationship between the Composite Learning Index and the Economic and Social Well-Being Index in Canada. 100 90 80 70 60 50 Policy message y = 0.7691x + 20.249 R 2 = 0.6979 40 40 50 60 70 80 90 100 Composite Learning Index
Environmental Sustainability Index ESI and Environmental Performance Index EPI. http://epi.yale.edu/home 42
Rank 1-10 Rank 11-20 Rank 21-30 Rank 31-40 Rank 41-50 Rank 51-60 Rank 61-70 Rank 71-80 Rank 81-90 Rank 91-100 Rank 101-110 Rank 111-120 Rank 121-130 Rank 131-140 Rank 141-149 Rank 1-10 Rank 11-20 Rank 21-30 Rank 31-40 Rank 41-50 Rank 51-60 Rank 61-70 Rank 71-80 Rank 81-90 Rank 91-100 Rank 101-110 Rank 111-120 Rank 121-130 Rank 131-140 Rank 141-149 2008 EPI: Uncertainty analysis for 149 countries Uncertainty & sensitivity analysis http://epi.yale.edu/ Home 43 Switzerland 31 30 20 11 6 Viet Nam 5 10 18 29 20 8 Sweden 63 25 10 Nicaragua 8 21 28 14 10 14 Norway 55 31 6 Saudi Arabia 11 13 23 16 10 11 Finland 81 16 Tajikistan 9 6 14 19 18 19 11 Costa Rica 81 16 Azerbaijan 10 23 9 24 26 5 Austria 15 19 16 21 18 Nepal 5 9 8 15 28 14 13 6 New Zealand 98 Morocco 14 20 15 13 15 16 Latvia 25 39 26 6 Romania 14 10 33 16 11 5 Colombia 74 18 5 Belize 9 29 16 13 14 6 France 15 26 30 14 13 Turkmenistan 5 6 14 8 19 25 11 9 Iceland 11 15 5 14 15 9 10 13 Ghana 11 14 16 10 10 13 9 8 5 Canada 58 33 9 Moldova 8 10 13 30 31 Germany 14 40 21 20 Namibia 16 16 25 18 16 United Kingdom 11 44 29 11 Trinidad & Tobago 6 20 11 23 8 13 8 5 Slovenia 9 18 25 23 8 10 Lebanon 5 15 13 13 5 5 6 20 8 11 Lithuania 16 20 14 9 8 9 6 9 Oman 10 25 18 24 5 10 Slovakia 15 21 14 25 5 6 8 Fiji 6 15 13 19 14 20 5 Portugal 20 46 16 11 5 Congo 9 23 18 13 13 13 9 Estonia 56 34 Kyrgyzstan 5 6 15 13 30 15 9 Croatia 16 19 23 10 6 9 5 5 5 Zimbabwe 21 11 15 10 15 13 6 Japan 6 38 35 14 5 Kenya 18 15 11 11 11 11 6 8 6 Ecuador 63 26 5 South Africa 14 16 20 11 16 11 Hungary 6 6 13 16 20 6 10 9 Botswana 5 13 18 18 19 16 6 Italy 6 28 24 16 13 5 Syria 16 11 21 13 30 6 Denmark 8 9 6 15 13 14 8 6 11 Mongolia 5 13 9 25 18 15 10 Malaysia 31 48 15 5 Laos 10 8 9 10 6 10 9 19 11 Albania 9 11 6 13 10 16 5 6 9 5 Indonesia 9 10 11 23 19 14 5 6 Russia 9 33 43 9 Côte d'ivoire 10 13 15 20 8 11 9 6 Chile 16 46 25 8 Myanmar 6 5 16 24 26 15 5 Spain 5 30 18 19 14 11 China 9 13 9 25 19 13 5 Luxembourg 9 15 16 20 26 5 5 Uzbekistan 14 16 29 29 11 Panama 73 20 Kazakhstan 5 15 16 36 24 Dominican Rep. 18 54 21 6 Guyana 6 10 19 15 20 20 5 Ireland 5 16 13 15 13 13 9 5 Papua New Guinea 6 10 14 9 11 20 20 8 Brazil 5 20 29 24 11 Bolivia 8 10 23 13 20 11 Uruguay 11 15 9 8 9 10 9 14 Kuwait 9 5 15 28 41 Georgia 8 8 19 15 16 10 13 5 United Arab Em. 10 9 36 19 11 11 Argentina 10 23 28 24 11 Tanzania 11 13 16 9 11 11 8 6 6 United States 5 23 19 24 13 8 Cameroon 6 10 6 13 23 23 15 Taiwan 20 13 19 16 10 13 Senegal 5 6 9 16 30 24 6 Cuba 5 24 29 19 13 5 Togo 8 6 18 18 18 18 10 Poland 5 11 20 35 15 Uganda 8 5 5 6 11 20 21 11 10 Belarus 11 10 10 18 16 16 13 Swaziland 6 16 31 24 15 Greece 8 18 14 19 15 5 10 6 Haiti 10 21 23 30 10 Venezuela 5 11 36 25 18 5 India 11 15 31 25 13 Australia 30 30 14 10 9 Malawi 9 13 13 14 11 15 9 6 8 Mexico 11 15 34 28 6 Eritrea 6 13 16 16 25 18 Bosnia & Herzegovina 5 10 11 24 9 6 8 14 6 Ethiopia 6 8 9 8 9 25 26 5 Israel 5 31 19 19 13 5 6 Pakistan 23 9 26 18 18 Sri Lanka 19 36 16 16 10 Bangladesh 9 18 24 48 South Korea 6 14 14 19 9 8 13 8 Nigeria 6 5 13 15 24 23 6 5 Cyprus 10 9 25 14 28 6 Benin 10 11 10 14 13 9 11 13 Thailand 8 30 35 11 11 Central Afr. Rep. 13 14 16 38 13 Jamaica 8 15 24 11 11 9 10 5 Sudan 10 34 46 6 Netherlands 9 11 14 10 21 9 11 9 Zambia 10 10 14 9 21 21 11 Bulgaria 5 19 25 15 8 10 6 Rwanda 6 11 18 11 18 5 13 6 9 Belgium 13 6 11 6 6 16 10 13 9 Burundi 9 8 15 9 18 29 11 Mauritius 6 9 19 18 8 16 15 Madagascar 8 13 16 20 21 15 Tunisia 5 10 10 10 14 19 18 9 Mozambique 6 6 9 11 14 18 21 9 Peru 15 30 18 30 Iraq 11 26 60 Philippines 6 13 26 21 16 9 5 Cambodia 8 15 11 31 28 Armenia 6 13 19 8 16 18 8 6 Solomon Islands 16 81 Paraguay 11 18 20 18 9 8 5 6 Guinea 13 14 23 36 6 Gabon 6 35 28 16 5 6 Djibouti 8 18 35 39 El Salvador 5 6 13 16 9 10 9 8 9 8 5 Guinea-Bissau 15 14 28 19 15 5 Algeria 5 5 15 26 24 11 6 5 Yemen 6 29 63 Iran 11 23 26 18 16 Dem. Rep. Congo 13 29 26 23 Czech Rep. 9 8 15 11 13 19 15 10 Chad 8 16 33 40 Guatemala 10 16 23 26 14 8 Burkina Faso 9 6 18 43 25 Jordan 8 14 24 20 6 14 8 Mali 5 18 36 41 Egypt 19 21 24 13 10 6 Mauritania 9 25 40 24 Turkey 18 15 18 16 9 6 6 Sierra Leone 11 18 70 Honduras 9 28 20 15 13 8 5 Angola 19 79 Fyrom 5 5 15 10 18 21 13 6 5 Niger 6 19 73 Ukraine 8 15 6 23 11 10 13 10 Legend: Probability between 5 and 15% Source: JRC calculations Probability between 15 and 30% Notes: Probability between 30 and 50% 1. Numbers express probabilities for the country rank Probability greater than 50% 2. Countries listed based on the 2008 EPI scores from highest to lowest (left to right) Probability lower than 5% is not shown
2004 Knowledge Economy Index (rank) Sweden Denmar Luxemb Finland USA Japan United Netherl Ireland Austria Belgium France EU15 EU27 German Sloveni Estonia Malta Cyprus Spain Czech Latvia Italy Greece Lithoua Hungar Portuga Slovaki Poland 0 5 10 15 Sensitivity analysis 20 25 30 Results The Knowledge Economy Index (FP6 - DG RTD) 44
What are the levels of Civic Competence of young people in Europe? 84 indicator index IEA data on: youth knowledge, skills, attitudes, values and beliefs towards citizenship Results: Newer democracies perform better on Attitudes towards participation and Citizenship values. Older and more stable democracies perform better on Social Justice and Cognitive tasks on civic 45 knowledge and skills.
0.08 0.18 0.19 0.23 0.23 0.24 0.25 0.25 0.26 0.26 0.27 0.29 0.31 0.33 0.33 0.35 0.36 0.36 0.36 0.37 0.44 0.45 0.47 0.47 0.48 0.48 0.49 0.50 0.53 0.55 0.57 0.59 0.60 0.61 0.62 0.64 0.67 0.73 2007 Summary Innovation Index COMPARATIVE ANALYSIS OF INNOVATION PERFORMANCE of THE EUROPEAN COUNTRIES Data from Eurostat Science and Technology Indicators and Community Innovation Survey (CIS) 0.80 0.70 0.60 0.50 0.40 0.30 0.20 DK SE NO FI US BE CA CH IL JP DE NL IE FR ES HR EL UK IS AT AU IT LU EE SI CZ MT HU SK PT CY PL BG RO LV LT 0.10 TR 0.80 THE 2007 SUMMARY INNOVATION INDEX (SII) 0.00-4.0-3.0-2.0-1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 Average grow th rate of SII (2003-2007) Sw eden Moderate innovators Innovation leaders Catching-up countries Innovation follow ers Turkey Dotted lines show EU performance. 0.70 0.60 0.50 0.40 0.30 0.20 0.10 Country scores, analysis of trends, and variability across countries and possible underlying reasons Nordic countries, Germany and UK are ahead of the US. 0.00 TRRO LV BGHR PLSK PT EL HULT MTES CY IT SI CZNOAUEECA EU BEFRNL AT IE IS LU US UK DEJP DK IL FI CH SE 46
Other sources: Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society, A 168(2), 307-323. Saltelli, A., 2007, Composite indicators between analysis and advocacy Social Indicators Research, 81, 65-77. Brand DA, Saisana M, Rynn LA, Pennoni F, Lowenfels AB, 2007, Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 4(4), 752-759. 47