Composite indicators. Andrea Saltelli EXPERT MEETING OECD WORKSHOP ON INDICATORS OF REGULATORY MANAGEMENT SYSTEMS

Transcription

1 Composite indicators Andrea Saltelli OECD WORKSHOP ON INDICATORS OF REGULATORY MANAGEMENT SYSTEMS EXPERT MEETING 2-3 April 2009 BERR CONFERENCE CENTRE LONDON, UNITED KINGDOM 1

2 Course based on: Joint OECD-JRC handbook. 5 years of preparation, 2 rounds of consultation with OECD high level statistical committee, finally endorsed March 2008 with one abstention 2

3 How many composites are around? Searching composite indicators on Scholar Google: October Scholar Google June ,440 May ,900 October ,030 February ,300 3

4 4 Analytic work based on CI s

5 See From Jean Philippe Cotis and Romain Duval, Competitiveness, Innovation and economic growth Istanbul 2007, Conference Measuring and Fostering Progress 5

6 See From Ludger Wößmann Contribution of Education and Training to Innovation and Growth Symposium on the Future Perspectives of European Education and Training for Growth, Jobs and Social Cohesion, Brussels,

7 7 Composite indicators and the media

8 Our rankings Business Economics Big Mac index: A light-hearted guide to whether currencies are at their correct level All Economics rankings» Education Living Politics Global peace index: The world's most and least peaceful countries All Politics rankings» Risk Technology Use: Rankings from the Economist web edition, March 30,

9 9 Economically literate press does have appetite for statistic based narratives.

10 Jean Pisani-Ferry and Andre Sapir argument on league tables 10

11 [ ] civil societies learn from the experience of others. Such policy learning can be enhanced by initiatives that facilitate cross country comparison and benchmarking. A telling example in this respect is [ ] PISA. 11

12 Transparency benefits the democratic process as it empowers national electorates to review the performance of their own governments and it helps focus the debate on key areas of underperformance. The use of league tables facilitates this process. 12

13 Tito Boeri and the World Bank s Doing Business report, speaking at the Italian Rai3, March

14 A ten-pillars snapshot of doing business in different countries 181 economies, from Afghanistan to Zimbabwe 14

15 15

16 Ranks for Italy Italy lost six position in a year, says Boeri, then he goes into the details. Paying taxes and enforcing contracts 16

17 International university rankings: How reliable are these rankings? THES World University Rankings What can we do to improve our position on the international scene? Jiao Tong ranking of World Universities 17

18 18 Can good practices help? Ten steps to build Composite Indicators

19 Ideally, a composite indicator should be based on: -a solid theoretical framework, -underlying data of good quality, -a defensible construct. 19

20 A good technical preparation for a CI can make it more robust (to uncertainties in data, weights, ) more resilient (remain relevant over time), more defensible (in dialogue with stakeholders ) 20

21 Step 1. Developing a solid theoretical framework What is badly defined is likely to be badly measured The challenges are: To integrate a broad set of (probably conflicting) points of view while keeping within a manageable construct To have a community of peers (individuals, regions, countries) willing to accept the theoretical framework. 21 Examples

22 Examples Doing Business provides a quantitative measure of regulations for starting a business, dealing with construction permits, employing workers, registering property, getting credit, protecting investors, paying taxes, trading across borders, enforcing contracts and closing a business [ ]. 22

23 Examples Doing Business does not measure all aspects of the business environment [ ]. It does not, for example, measure security, macroeconomic stability, corruption, the labor skills of the population, the underlying strength of institutions or the quality of infrastructure [ ]. 23

24 The Canadian Council on Learning has developed the Composite Learning Index ( The framework: Learning to Know Learning to Do Learning to Live Together Learning to Be[*] Jacques Delors **+ Pillars from Jacques Delors Task Force: UNESCO's International Commission on Education for the Twenty-first Century. 24

25 I. Learning to Know involves developing the foundation of skills and knowledge needed to function in the world. This includes literacy, numeracy, general knowledge and critical thinking. II. Learning to Do refers to the acquisition of applied skills. It can encompass technical and hands-on skills and knowledge, and is closely tied to occupational success. 25

26 III. Learning to Live Together involves developing values of respect and concern for others, fostering social and inter-personal skills, and an appreciation of the diversity of Canadians. This area of learning contributes to a cohesive society. IV. Learning to Be refers to the learning that helps develop the whole person mind, body and spirit. This aspect concerns personal discovery, self-knowledge, creativity and achieving a healthy balance in life (~Maslow s top). 26

27 Step 1. Developing a solid theoretical framework After Step 1. the developer should have A clear understanding and definition of the multidimensional phenomenon to be measured. A nested structure of the various sub-groups of the phenomenon. A list of selection criteria for the underlying indicators, e.g., input, process, output. 27

28 Step 2. Selecting indicators A composite indicator is above all the sum of its parts Excerpt: The strength of a composite indicator can largely depend on the quality of the underlying data. * +. The theoretical framework should guide the choice of the underlying indicators. The selection process can be quite subjective and therefore should involve stakeholders. Moreover, depending on availability of data, certain indicators cannot be used and proxies need to be considered. 28

29 Eigenvalues Step 3. Multivariate analysis Analysing the underlying structure of the data is an art a1 1 a a a a a b b b b b b a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b Component number 29 Component Initial Eigenvalues Total % of variance Cumulative %

30 Step 4. Imputation of missing data. The idea of imputation could be both seductive and dangerous Almost all datasets contain missing data. Country a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6 AT BE BG CY CZ DE DK EE EL ES EU FI FR

31 Three common approaches to deal with missing data: case deletion (removes either country or indicator from the analysis) single imputation (e.g. Mean/Median substitution, Regression, etc.) multiple imputation (e.g. Markov Chain Monte Carlo algorithms). Trying to minimize bias and to keep expensive to collect data that would otherwise be discarded. 31

32 Step 5. Normalisation of data Avoid adding up apples and oranges Ranking Standardization Re-scaling Distance to reference country Categorical scales Cyclical indicators Balance of opinions 32

33 Step 6. Weighting and aggregation The relative importance of the indicators can become the substance of a negotiation Weights based on statistical models Principal component/factor analysis Data envelopment analysis Regression approach Unobserved components models 33

34 Step 6. Weighting and aggregation The relative importance of the indicators can become the substance of a negotiation Weights based on opinions: participatory methods Budget allocation Public opinion Analytic hierarchy process Conjoint analysis 34

35 Step 7. Robustness and sensitivity Uncertainty analysis can be used to assess the robustness of composite indicators Space of alternatives Performance index 60 Weights Imputation Aggregation Including/ Normalisation excluding variables Spain Italy Greece 35

36 Economic and Social Well-being Step 8. Links to other variables Composite indicators can be linked to other variables and measures Comparing effectively complex dimensions: Figure 1. Relationship between the Composite Learning Index and the Economic and Social Well-Being Index in Canada y = x R 2 = Composite Learning Index 36

37 Step 9. Back to the details De-constructing composite indicators can help extend the analysis Ad hoc clustering 37

38 Step 10. Presentation and dissemination A well-designed graph can speak louder than words The four-quadrant model of the Composite Learning Index (Canadian Council on Learning) 38

39 39 In house CI development

40 Concept: (WHO report) Published in PLoSMedicine Sensitivity analysis Results Policy message 40 The Alcohol Policy Index (New York Medical College)

41 Economic and Social Well-being CCL Report Framework Sensitivity analysis Scenario Pillar Structure Normalisation Weighting Aggregation CLI Preserved z-scores FA within pillar, Regression Linear weights to Factors, FA pillars, Regression weights to pillars S1 Preserved z-scores FA within pillar, FA pillars Linear S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric S12 Preserved Min-max FA within pillar, FA pillars Geometric S13 Not preserved z-scores FA all indicators Geometric S14 Not preserved Min-max FA all indicators Geometric S15 Preserved z-scores FA within pillar, EW pillars Geometric S16 Preserved Min-max FA within pillar, EW pillars Geometric S17 Not preserved z-scores EW all indicators Geometric S18 Not preserved Min-max EW all indicators Geometric S19 Preserved z-scores EW within pillar, EW pillars Geometric S20 Preserved Min-max EW within pillar, EW pillars Geometric S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria S22 Not preserved Raw data FA all indicators Multi-criteria S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria S24 Not preserved Raw data EW all indicators Multi-criteria S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria 41 Results The Composite Learning Index (Canadian Council on Learning) Figure 1. Relationship between the Composite Learning Index and the Economic and Social Well-Being Index in Canada Policy message y = x R 2 = Composite Learning Index

42 Environmental Sustainability Index ESI and Environmental Performance Index EPI. 42

43 Rank 1-10 Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank 1-10 Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank Rank EPI: Uncertainty analysis for 149 countries Uncertainty & sensitivity analysis Home 43 Switzerland Viet Nam Sweden Nicaragua Norway Saudi Arabia Finland Tajikistan Costa Rica Azerbaijan Austria Nepal New Zealand 98 Morocco Latvia Romania Colombia Belize France Turkmenistan Iceland Ghana Canada Moldova Germany Namibia United Kingdom Trinidad & Tobago Slovenia Lebanon Lithuania Oman Slovakia Fiji Portugal Congo Estonia Kyrgyzstan Croatia Zimbabwe Japan Kenya Ecuador South Africa Hungary Botswana Italy Syria Denmark Mongolia Malaysia Laos Albania Indonesia Russia Côte d'ivoire Chile Myanmar Spain China Luxembourg Uzbekistan Panama Kazakhstan Dominican Rep Guyana Ireland Papua New Guinea Brazil Bolivia Uruguay Kuwait Georgia United Arab Em Argentina Tanzania United States Cameroon Taiwan Senegal Cuba Togo Poland Uganda Belarus Swaziland Greece Haiti Venezuela India Australia Malawi Mexico Eritrea Bosnia & Herzegovina Ethiopia Israel Pakistan Sri Lanka Bangladesh South Korea Nigeria Cyprus Benin Thailand Central Afr. Rep Jamaica Sudan Netherlands Zambia Bulgaria Rwanda Belgium Burundi Mauritius Madagascar Tunisia Mozambique Peru Iraq Philippines Cambodia Armenia Solomon Islands Paraguay Guinea Gabon Djibouti El Salvador Guinea-Bissau Algeria Yemen Iran Dem. Rep. Congo Czech Rep Chad Guatemala Burkina Faso Jordan Mali Egypt Mauritania Turkey Sierra Leone Honduras Angola Fyrom Niger Ukraine Legend: Probability between 5 and 15% Source: JRC calculations Probability between 15 and 30% Notes: Probability between 30 and 50% 1. Numbers express probabilities for the country rank Probability greater than 50% 2. Countries listed based on the 2008 EPI scores from highest to lowest (left to right) Probability lower than 5% is not shown

44 2004 Knowledge Economy Index (rank) Sweden Denmar Luxemb Finland USA Japan United Netherl Ireland Austria Belgium France EU15 EU27 German Sloveni Estonia Malta Cyprus Spain Czech Latvia Italy Greece Lithoua Hungar Portuga Slovaki Poland Sensitivity analysis Results The Knowledge Economy Index (FP6 - DG RTD) 44

45 What are the levels of Civic Competence of young people in Europe? 84 indicator index IEA data on: youth knowledge, skills, attitudes, values and beliefs towards citizenship Results: Newer democracies perform better on Attitudes towards participation and Citizenship values. Older and more stable democracies perform better on Social Justice and Cognitive tasks on civic 45 knowledge and skills.

46 Summary Innovation Index COMPARATIVE ANALYSIS OF INNOVATION PERFORMANCE of THE EUROPEAN COUNTRIES Data from Eurostat Science and Technology Indicators and Community Innovation Survey (CIS) DK SE NO FI US BE CA CH IL JP DE NL IE FR ES HR EL UK IS AT AU IT LU EE SI CZ MT HU SK PT CY PL BG RO LV LT 0.10 TR 0.80 THE 2007 SUMMARY INNOVATION INDEX (SII) Average grow th rate of SII ( ) Sw eden Moderate innovators Innovation leaders Catching-up countries Innovation follow ers Turkey Dotted lines show EU performance Country scores, analysis of trends, and variability across countries and possible underlying reasons Nordic countries, Germany and UK are ahead of the US TRRO LV BGHR PLSK PT EL HULT MTES CY IT SI CZNOAUEECA EU BEFRNL AT IE IS LU US UK DEJP DK IL FI CH SE 46

47 Other sources: Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society, A 168(2), Saltelli, A., 2007, Composite indicators between analysis and advocacy Social Indicators Research, 81, Brand DA, Saisana M, Rynn LA, Pennoni F, Lowenfels AB, 2007, Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 4(4),