Interim Evaluation of the Lifelong Learning Programme ( ) ANNEXES TO THE FINAL REPORT

Transcription

1 EUROPEAN COMMISSION Directorate-General for Education and Culture Service Contract No PPMI Interim Evaluation of the Lifelong Learning Programme ( ) ANNEXES TO THE FINAL REPORT 18 February 2011 Contractor:

2 TABLE OF CONTENTS TABLE OF CONTENTS... 2 LIST OF ACRONYMS... 4 ANNEX 1. TERMS OF REFERENCE... 6 ANNEX 2. SYNTHESIS OF THE NATIONAL REPORTS... 7 Introduction Relevance of the lifelong learning programme Relevance of the objectives Community Added Value Complementarities between the LLP and other programmes Effectiveness of the Lifelong Learning Programme The level and quality of implementation Integration of the LLP Reaching the target groups Contribution to the EU priorities Horizontal policies Dissemination and exploitation of results Visibility of the LLP Efficiency of the lifelong learning programme Cooperation between the main implementation bodies, management workload and efficiency of monitoring mechanisms Divided management responsibilities for centralised and decentralised actions Management supporting tools Adequacy of resources ANNEX 3. CASE STUDIES Case study template Introduction Case study framework Interview questionnaires Interview questionnaire for national institutions and national social partners Interview questionnaire for the institutional beneficiaries EU-level case studies Programme-Policy Link Dissemination and Exploitation of the Lifelong Learning Programme Results Transversal Key Activity 2 Languages Jean Monnet KA1 Excellence Centres and Chairs Jean Monnet KA2 Operating Grants to Support Specified Institutions Erasmus Academic Networks National case studies Erasmus Student Mobility for Studies (Outgoing Students): Lithuania Erasmus Student Mobility for Studies (Incoming Students): Germany

3 9. Erasmus Staff Teaching Abroad : the Czech Republic Erasmus Staff Teaching Abroad : Norway Comenius Bilateral and Multilateral Partnerships : Italy Comenius Assistantship : Bulgaria Leonardo da Vinci Professionals in Vocational Education and Training (VETPRO): the Netherlands Leonardo da Vinci Transfer of Innovation : The Netherlands Grundtvig Learning Partnership : Slovenia Grundtvig In-service Staff Training for Adult Education Staff : Denmark ANNEX 4. LIST OF INTERVIEWS ANNEX 5. SURVEY RESULTS Survey of the National Authorities and National Agencies involved in the implementation of the LLP Survey of institutional beneficiaries Survey of individual beneficiaries ANNEX 6. THE LONGITUDINAL MAPPING OF THE LIFELONG LEARNING PROGRAMME Comparison of the Previous Programmes and the Lifelong Learning Programme ANNEX 7. ANALYSIS OF THE MONITORING DATA ANNEX 8. NOTE ON MONITORING THE LIFELONG LEARNING PROGRAMME Introduction Description of the current situation The need for and use of monitoring information (internal/management needs and external/accountability needs) Areas for possible improvement and strengthening of the baseline situation An outline for a monitoring system of the current programme and for the successor programme ANNEX 9. LIST OF REFERENCES AND SOURCES

4 LIST OF ACRONYMS AED AIA BMBF BSP DAAD CHES CIFE CMEPIUS D&E DG EAC DZS EACEA EAN EC ECTS EESF EIPA EO ERA EU EUC EUI HE HRDC ICT IO/AAAs IVT KA KTU LEI/LEIs LMTA LLP LP MES MS MP / MPs MSP MoEYS NA NAE NAEP NAVET OMC The European Agency for Development in Special Needs Education Academic Information Agency Federal Ministry of Education and Research Bilateral School Partnership German Academic Exchange Service Centre for Higher Education Studies The International Centre for European Training Centre for mobility in Slovenia - NA Dissemination and exploitation The Directorate General for Education and Culture House of International Services MoEYS (Dům zahraničních styků MŠMT) The Education, Audiovisual and Culture Executive Agency Erasmus Academic Networks European Commission European Credit Transfer and Accumulation System Education Exchanges Support Foundation (Lithuania) The European Institute of Public Administration Equal opportunities The Academy of European Law European Union Erasmus University Charter European University Institute Higher education Human Resource Development Center Information and Communication Technologies International Offices at the Universities Initial Vocational Training Key Activity Kaunas Technology University Local ERASMUS Student Initiative Lithuanian Music and Theatre Academy Lifelong Learning Programme Learning Partnership Ministry of Education and Sports Member State Multilateral project / projects Multilateral School Partnership Ministry of Education, Youth and Sport National agency National ERASMUS expert National Agency for Educational Programmes National Agency for Vocational Education and Training Open Method of Coordination Leonardo da Vinci People in the Labour Market 4

5 PLM PPMI PR SF SIU SMS SMP STA ToI U3O VET VETPRO VGTU Public Policy and Management Institute Public Relations Structural funds Senter for internasjonalisering av høgre utdanning Norwegian National Agency Students mobility for studies Students mobility for placements Staff Teaching Abroad Transfer of Innovation University for third age Vocational education and training Leonardo da Vinci Professionals in Vocational Education and Training Vilnius Gediminas Technical University Euro (currency) ABBREVIATIONS OF THE STATES: AT BE BE(fr) BE(nl) BG CY CZ DE DK EE EL ES FI FR HU IE IS IT LT LU MT NL NO PL PT RO SE SI SK TR UK Austria Belgium The French-speaking Community of Belgium The Flemish Community of Belgium Bulgaria Cyprus The Czech Republic Germany Denmark Estonia Greece Spain Finland France Hungary Ireland Iceland Italy Lithuania Luxembourg Malta The Netherlands Norway Poland Portugal Romania Sweden Slovenia Slovakia Turkey The United Kingdom 5

6 ANNEX 1. TERMS OF REFERENCE TERMS OF REFERENCE Tender no. EAC/32/2009 INTERIM EVALUATION OF THE LIFELONG LEARNING PROGRAMME Contracting Authority: European Commission, Directorate-General for Education and Culture Provided in a separate document 6

7 ANNEX 2. SYNTHESIS OF THE NATIONAL REPORTS Introduction Following article 15.4 of the LLP Decision No. 1720/2006, the Member Countries submitted to the Commission national reports on the implementation of the Lifelong Learning Programme (LLP) in This document provides a synthesis of the national reports. The main objective is to provide a coherent narrative of the views regarding the success of the implementation of the LLP in the Participating Countries. The synthesis aims to outline the key areas, where the opinions expressed in the national reports converge, as well as the particularities and differences between the Participating Countries. The Synthesis is an important source of information for the purposes of interim evaluation of the LLP. However, the findings provided in this document should be treated with caution. The national reports differed in their structure, adopted methodologies, scope and the level of analysis. Therefore, while the general conclusions reflect the views of all national reports, the discussion of concrete issues and recommendations reflect only the views of specific Participating Countries. The structure of the synthesis follows the template that the Commission provided for preparation of the national reports. Whenever relevant, the synthesis discusses the specificities of each sectoral programme separately. 1. Relevance of the lifelong learning programme 1.1. Relevance of the objectives To what extent do the LLP objectives remain relevant/contribute to national policy priorities, lifelong learning strategies and socio-economic needs? Overall, the Participating Countries argue that the LLP objectives are relevant to the national policy priorities, lifelong learning strategies and socio-economic needs. In fact, the LLP objectives are either explicitly integrated into the national policy documents (for instance, in CZ, EE, HU, LT, RO and SI among others) or implicitly recognised as key areas for further development of national education and training systems. The most relevant objectives of the Programme include higher mobility and internationalisation of education and training systems. Usually, internationalisation is understood as international cooperation and the establishment of partnerships among educational institutions. Hence, internationalisation is closely linked with improving the quality of education and training, increasing openness and ability to work in intercultural environments. The LLP is seen as one of the key drivers of internationalisation of education and training in a large number of countries (e.g. including AT, BE(nl), EE, FI, FR, NL, NO, PT, SE). However, several Participating Countries point out that the LLP objectives do not fully match the national needs in the area of education and training. For instance, the representatives from Poland argue that higher priority should be given to language learning (particularly in the area of adult education). The UK points out that the LLP objectives do not fully reflect the increasing national focus on improving learners skills and employability, which are also emphasised in EU 2020 Strategy. In the national report from France, it is argued that both national and 7

8 European priorities of Comenius are not adjusted to the beneficiaries needs, e.g. pupils and teachers mobility, acquisition of new skills etc. Other countries (e.g. Austria, the Netherlands and Portugal) argue that while the objectives of the overall LLP are relevant, they do not adequately take into consideration the most pressing national socio-economic needs. The following paragraphs concentrate on relevance of each sectoral programme. Comenius. Overall, the objectives of the Comenius sectoral programme are relevant in the Participating Countries. A vast majority of the countries argue that objectives are in the line with national priorities or needs. The importance of the objectives depends on national contexts, however, the majority of the countries emphasise three objectives are particularly relevant: 1) the improved quality of education and training; 2) higher accessibility to the learning opportunities; 3) learning of foreign languages. Improvements in the quality of education is particularly emphasised by BE (nl), BG, CY, CZ, EE, ES, HU, IT, LT, NO, PL and TR. The national reports argue that this objective could be attained via: a) exchange of good practice; b) application of innovations; c) promotion of creativity and competitiveness as well as acquisition of basic-life skills and competences. Attainment of the latter three operational objectives is closely associated with the following LLP outputs: a) higher mobility; b) increased volume of school partnerships; c) improvements in multidisciplinary pedagogical approaches. Special importance is also given to the in-service training of teachers and educational staff by Cyprus, Estonia, the Flemish Community of Belgium, Ireland and Sweden among others. Higher accessibility to the learning opportunities is another highly relevant objective, which is explicitly pointed out by AT, BG, CY, EE, ES, FI, IS, LT, NL, PL, UK. Universally, higher accessibility is seen as important for the people with special needs and disadvantaged groups. In addition, some specific target groups are also accentuated: migrants (e.g. Austria, Finland and Poland), early school leavers (e.g. Cyprus, Iceland and Poland), people living in rural areas (e.g. Estonia and Spain) and ethnic minorities (e.g. Cyprus and the UK). Learning of foreign languages, linguistic diversity and the development of common understanding of the diversity of cultures is the third highly relevant objective (e.g. in AT, BE (nl), BG, ES, FI, IT, LT, NL, NO, PL and TR). It also contributes to the better quality of education system, accessibility to the learning opportunities and personal development. The learning of foreign languages is especially prioritised in the French-speaking Community of Belgium, Cyprus and Portugal, as hindrances to mobility are created there due to insufficient language competencies. Finally, several countries also emphasise the importance of such objectives as ICT-based learning (e.g. BG, CZ, ES, LT and PL), contribution to social cohesion, active citizenship and creation of a sense of European citizenship (e.g. BE (nl), CY, DE and IT). Erasmus. Generally, none of the countries declare that the Erasmus objectives are incompatible with the national policy priorities. Moreover, several objectives are particularly relevant for the majority of countries: 1) improved quality of education and training; 2) promotion of employability; 3) learning of foreign languages. Improved quality of education and training could be treated as the most relevant objective, which is outlined by the majority of the Participating Countries (e.g. AT, BE (nl), BG, CY, CZ, EE, ES, FI, FR, HU, IE, IT, LT, NO, PL, PT, SE, TR and UK). The Participating Countries see a clear link between improvements in quality and attainment of the following lower-level objectives: a) increased volume of student and teaching staff mobility (interestingly, in Luxembourg, students mobility became compulsory since 2006/2007); b) increased volume of multilateral cooperation between higher education institutions (in Portugal, for example, cooperation is seen as means of attracting qualified students from different scientific domains); c) the development, exchange and application of innovative practices in education and training, which contribute to the development of professional competence 8

9 of educational staff. Commonly, the countries assume that attainment of these objectives should also contribute to the promotion of the European dimension. Promotion of employability of students is another highly relevant objective for the Participating Countries (e.g. BE (nl), CY, EE, ES and FI). The national reports argue that Erasmus contributes to employability by facilitating the development of language and professional competences, as well as ability to work in an intercultural environment and improved capacity to adapt in different cultural contexts. Learning of foreign languages and linguistic diversity is also given high priority in the national reports (e.g. in BE (fr), BG, CY, ES, IE and IT). Leonardo da Vinci. The objectives of this sectoral programme are pertinent to the national needs. The most relevant issues tackled by Leonardo da Vinci can be divided into four major groups: 1) improved quality of vocational education and training (VET); 2) promoted employability and participation in the European labour market; 3) higher accessibility to the learning opportunities; 4) enhanced attractiveness of vocational education and training. Improved quality of VET is stressed as an important need in the national reports from BE(nl), CY, CZ, EE, FI, HU, IE, IT, LT, LU, NL, NO, PL, PT, SE and TR. The reports point out that increases in the quality of VET crucially depend on the following activities funded by the LLP: a) mobility of VET students and teachers; b) cooperation between the providers of VET, enterprises and social partners; c) development, transfer and application of the innovative practices. Promotion of employability and participation in the European labour market is another highly relevant objective. BE (nl), CY, CZ, ES, IE, NO, PT, SE and TR among others argue that employability could be facilitated by acquiring and using innovative knowledge, skills and qualifications. Mobility is seen as particularly important for enhancing employability, since it expands the opportunities for students placement and traineeships. Higher accessibility to the learning opportunities is also seen as a relevant issue for AT, BE (nl), CY, FI, LT, LU and NO among others. Generally, it encompasses people of all ages, those with special needs and disadvantaged groups. Additionally, some countries also accentuate the migrants (e.g. Austria and Finland), social, economic and institutional stakeholders (e.g. Italy), members of non-profit, non-governmental and voluntary organisations (e.g. the UK), and unemployed people (e.g. Norway and Poland) as distinctive target groups. Enhancing attractiveness of vocational education and training is the fourth highly relevant objective. The national reports from BE (nl), CY, CZ, DE, LT, NO and PL notice the declined interest in vocational education and training. Thus, promotion of attractiveness is a vital element of Leonardo da Vinci. Three previously discussed objectives (improved quality, promoted employability and higher accessibility to the learning opportunities) could significantly contribute to enhanced attractiveness of Leonardo da Vinci. Grundtvig. The objectives of the Grundtvig sectoral programme are coherent with the national priorities and socio-economic needs. While issues are varying according to the national contexts, four objectives were particularly emphasised in the national reports: 1) higher accessibility to the learning opportunities; 2) improved quality of education and training; 3) promotion of non-formal and informal learning; 4) learning of foreign languages. Higher accessibility to the learning opportunities is an exceptionally valued objective, as it is emphasised in the reports from AT, BE (nl), BG, CY, CZ, ES, IS, IT, LT, NL, NO, PL and TR. Higher accessibility is associated with the two objectives: a) assisting the vulnerable social groups (elderly and those without basic qualifications); b) providing them with pathways to improving their knowledge and competencies. Additionally, other specific target groups are also pointed out: migrants (e.g. Austria, Cyprus, the Flemish Community of Belgium and Spain), refugees (e.g. Cyprus and the Flemish Community of Belgium), young leavers (e.g. Bulgaria, Iceland and 9

10 Spain), prisoners (e.g. the Flemish Community of Belgium), adults with mental disabilities (e.g. Cyprus and the Flemish Community of Belgium) unemployed persons (e.g. Italy) and learners from deprived areas (e.g. the Netherlands). Improved quality of education and training is another highly relevant objective. It is strongly linked to the development, transfer and application of innovative practices in education and training (e.g. BE (nl), BG, CY, EE, ES, HU, LT, NO, PL and TR) as well as improvements in pedagogical competencies and approaches (including BE (nl), BG, CY, EE, FI, LT, NO, PL, SE and TR). Higher levels of non-formal and informal education are particularly pertinent to the national needs, although these forms of education are not explicitly recognised by the operational objectives of Grundtvig (see the national reports from BE (nl), CY, IT, LT, NO and PL). It is argued that generally Grundtvig is the only accessible possibility to improve competences for those adults, who are not covered by the formal education programmes. Learning of foreign languages is strongly emphasised in the national reports from BE (fr), BG, LT, NL, NO and PL. For instance, the Polish national report argues that currently the Grundtvig programme does not sufficiently prioritise adults language learning needs. Since, Grundtvig target groups generally lack foreign language skills, their participation in the programme is rather limited. Finally, several countries also emphasise relevance of such objectives as ICT-based learning (e.g. BE(nl), BG, CY, IS, LT and PL), employability and flexibility in the labour market (e.g. the Czech Republic, Finland, the Flemish Community of Belgium, Norway and Spain). Summary and recommendations To sum-up, there is a consensus among the Participating Countries that the objectives of the LLP remain relevant to the national policy priorities, lifelong learning strategies and socio-economic needs. Due to different national contexts, the opinions of the Participating Countries slightly diverge regarding the objectives, which are the most relevant. However, the largest proportion of the reports emphasise internationalisation and higher quality of education and training as the most important objectives of the LLP. In addition, the largest proportion of the reports emphasised the importance of the following objectives: Higher accessibility to the learning opportunities and learning of foreign languages are seen as very important Comenius objectives. Promotion of employability and learning of foreign languages were emphasised among the Erasmus objectives. Promotion of employability and participation in the European labour market, higher accessibility to the learning opportunities and enhanced attractiveness of vocational education and training stand out from the Leonardo da Vinci objectives. Higher accessibility to the learning opportunities, promotion of non-formal and informal learning and learning of foreign languages are very important Grundtvig objectives. The national reports also provided several recommendations regarding further improvements in the intervention logic of the LLP: The number of the LLP objectives and actions should be revised and reduced. Several Participating Countries argue that the focus of the Programme is rather blurred, which has negative repercussions of its effectiveness and efficiency (e.g. France, Finland and Poland). This view is backed by a report from the UK, which argues that the objectives should be streamlined so as to focus on key areas, for instance the quality of learning mobilities. Furthermore, according to the Flemish Community of Belgium, Hungary and Slovenia, the LLP goals are too general and not attainable due to short period of time (until 2013) and due to insufficient financial resources. Reports from BE (nl), DK, FI, FR, LU, NO, SE and the UK also argue that the number of various actions should be reduced. For instance, the national report from France suggests that the actions should be concentrated on four 10

11 major issues: preparatory visits, actions of mobility, exchange visits, development projects or transfer of innovations. Slovenia proposes only two main blocks the projects and mobilities. Poland recommends the revision and reformulation of the Programme objectives, which are not fully in the line with current economic and labour situation. the Flemish Community of Belgium authorities add that the LLP objectives should be simplified, it should have fewer partitions and actions, it should include more opportunities for cooperation and cross-fertilisation amongst different target groups <...> This should all be part of an exercise to rationalise and streamline the programme, with a focus on quality rather than quantity. Accessibility to the learning opportunities should be more strongly emphasised. Higher accessibility should encompass different age groups, people in different socio-economic positions, those with special needs (with disabilities, vision and hearing impaired persons) and migrants among others. The implementation of this goal would contribute to higher social inclusion and equality of opportunities (e.g. see the national reports from BG, DK, FI, IE, IT, LU, SI, TR and UK). Consider increasing the leverage of National Authorities and Agencies in setting national priorities and criteria for the selection of applications. Reports from the Czech Republic, Estonia, Iceland and the Netherlands argue that the national education and training systems considerably differ. Hence, there is a need for more flexibility in reacting to the most pressing domestic challenges Community Added Value What is the Community Added Value of the Programme? Does it effectively promote cooperation between Participating Countries? Overall, the national reports argue that the European added value of the LLP is high or very high. The Programme is also very successful in promoting cooperation between the Participating Countries. As the report from Bulgaria summarises: The added value is the aspiration for joint results and products, in an environment, which does not promote competition, but cooperation and exchange of experience and good practices. The largest European added value has been at the individual and organisation levels. Nevertheless, implementation of the LLP also has an impact at systemic level by facilitating clarity and transparency of the education and training systems across the Europe. Considerable added value of the Programme is found in the following areas: Long-term sustainable international cooperation and higher mobility (e.g. AT, BE (fr), BE (nl), CZ, DE, DK, ES, HU, IS, LT, LU, MT, NL, NO, PL, RO and UK). The cooperation is particularly important at the individual level (e.g. in developing new long-term partnerships between people engaged in similar activities) and organisational level (e.g. in facilitating cooperation between ministries, their partners, educational institutions, National Agencies etc.). Acquisition of new professional skills, intercultural competencies and knowledge of innovative work methods (e.g. AT, BE (fr), BG, CZ, DE, DK, ES, HU, IS, LU, PL, RO and SI). Improved knowledge about culture and foreign languages of the other European countries (e.g. AT, BG, CZ, DK, ES, FI, IT, NO, PL, RO, SI and TR). Personal maturity and development of mobile learners. This includes the development of tolerance and openness towards other people and viewpoints, overcoming stereotypes and prejudices, encouraging independency, self-esteem, a sense of communality and responsibility (e.g. in BG, CY, DK, ES, FI, IT, PL, SI and TR among others). Strengthening of the European dimension. This encompasses development of European identity, recognition of common European values and promotion of regional cohesion (e.g. AT, DE, DK, ES, IS, LT, MT, RO, SI). 11

12 Improved profile and image of the institution at a national level (due to enhanced transparency of qualifications), which contributes to the increased attractiveness of education (e.g. the French-speaking Community of Belgium, Denmark, Poland, Spain and Turkey). Long-lasting partnerships and contacts (e.g. BG, DE, ES, FI, FR, IT, LT, NL, NO, RO, SI, TR). Stronger interactions between educational and business sectors and promoted employability. According to the national report from Turkey, Erasmus Student Mobility for Placements brings together the education sector and business community to the set up a dialogue, helps graduated students to meet business qualifications, helps finding a work and facilitates quick job adaptation. Denmark, Italy and Spain among others also support this statement. Improved basic skills (especially vocational ones) of socio-economically disadvantaged beneficiaries and those with special needs (e.g. BG, ES, FI, IT, NO and TR among others). While the majority of the Participating Countries agree that the aforementioned benefits of the LLP are very important, opinions diverge regarding the extent, to which these benefits could be captured without the EU level intervention. On the one hand, a number of the Participating Countries argue that, if the Programme did not exist, a vast majority of people would not have a possibility to participate in mobility actions, and organisations would be deprived of opportunities for international cooperation due to insufficient financial resources, lack of information and other factors. As Finland summarises: < > without the programme many of its target groups would have much less possibilities for this level of international cooperation. Turkey adds: If the Grundtvig programme did not support and encourage people to implement projects in the international domain, almost all participants < > stated that their financial state, and < > their social environment would not allow them to perform such an activity. The national reports from Iceland and Romania also support these statements. On the other hand, the report from the Netherlands argues that some beneficiaries are already involved in international cooperation, which lasts for a considerable period of time. Such cooperation is likely to persist, even if the Programme was discontinued. As the Dutch report argues in the case of Erasmus, the structures and partnerships < > would remain active, if there was no Erasmus. Also, mobility would still exist, but in lower numbers. This, however, does not necessarily imply that the European added value of the LLP is low. Instead, such finding could be interpreted as evidence of sustainability of the results of the LLP and its predecessors. Table 2.1 illustrates the breakdown of the European added value by the sub-programmes. Table 2.1. Areas of the largest European added value resolved according to the sub-programmes. The sub-programmes of the LLP No. / Areas of the largest European added Leonardo Comenius Erasmus Countries value da Vinci Grundtvig 1. Long-term sustainable international cooperation and higher mobility Countries FI, FR, IT, LT, ES, FI, FR, IT, LT, FI, LT, NL, NO, NL, SI NL, TR, UK TR FI, NL, TR 2. Acquisition of new professional skills, intercultural competencies and knowledge of innovative work methods Countries BG, EL, ES, FI, GR, IT, LT, NO, TR, SI BG, ES, FI, RO, TR BG, FI, NO, RO, TR FI, NL, RO, TR 3. Learning of foreign languages Countries BG, ES, IT, LT, ES, TR TR NO, TR 4. Improved knowledge about other + + European cultures Countries BG, FI, IT, LT, BG, ES TR 5. Personal maturity and development

13 No. / Countries Countries Areas of the largest European added value of learners Comenius BG, ES, FI, IT, SI, TR The sub-programmes of the LLP Leonardo Erasmus da Vinci Grundtvig DK, ES, LT, TR 6. Strengthening of the European dimension + Countries DK, ES, FI, TR 7. Long-lasting partnerships and contacts Countries FI, IT, LT, NO BG, FI FI, LT, NO TR 8. Stronger interactions between educational and business sectors and promoted employability + + Countries ES, LT, TR BG, RO, TR 9. Improved basic skills of socioeconomically disadvantaged beneficiaries and those with special + needs Countries BG, EL 10. Improved image and attractiveness of the institution due to enhanced + transparency of qualifications Countries ES, RO, TR, UK Notes: + marks those sub-programmes, in which particular areas of the European added value were explicitly pointed out. Summary and recommendations To sum-up, the majority of the Participating Countries argue that the added value of the LLP is high. However, an overwhelming plurality of the reports do not analyse the merits of the European level intervention (in contrast to the national level efforts) and the extent to which the positive effects would occur in the absence of the Programme Complementarities between the LLP and other programmes How does the LLP complement and add value to other national and international programmes available in your country? Overall, the LLP strongly complements national and international programmes implemented in the area of education and training. The LLP is the main cross-border cooperation and mobility programme that supports initiatives covering all stages of learning from cradle to grave. The Participating Countries also argue that the sectoral programmes fill in important gaps, which are not covered by the national education policies (e.g. in the French-speaking Community of Belgium, Malta, the Netherlands, Poland and the UK). Several factors differentiate the LLP from other similar national and international programmes. First, the LLP offers mobility and cooperation opportunities to a wide range of target groups, while other similar programmes usually have considerably narrower approach. Second, the LLP provides opportunities for internationalisation and cooperation among a large number of countries. This could not be achieved by the other existing programmes. For example, the Nordplus Programme covers only the Northern European countries, which limits its geographical scope. The value of the LLP in the context of other implemented actions is associated with the fact that it complements other programmes, directed at: 13

14 Consistent cooperation, mobility promotion and international dimension of education and training strengthening activities (in EE, EL, ES, DE, LT, PT and UK). Improving the quality of education, sharing good experience and introducing innovations (e.g. BG, CZ, EE, EL, DE, LT, MT and UK among others). Promoting a more effective management of education institutions (e.g. Bulgaria among others). However, several reports argue that the Programme nearly overlaps with other related programmes, but it does not complement them (see the national report from Slovenia, to some extent the French-speaking Community of Belgium (as long as it concerns Leonardo da Vinci)). Even if some complements are observed, they are unsystematic, and, as a result, have a weaker impact (e.g. national master programmes in Slovenia). The main difficulties, which hinder the aforementioned benefits, are: Insufficient financial resources, which automatically limit the scope of the Programme implementation (e.g. Ireland, Lithuania). Administrative obstacles in, e.g. accommodating the LLP with national rules on public spending and the procedures of the European Cohesion Fund (e.g. Slovenia). A lack of a systematic and coherent approach at the national level, which hinder the added value (e.g. Slovenia). The following paragraphs concentrate on each sectoral programme. Comenius complements the existing programmes as well as occupies specific niches, which are not covered by the other national, bilateral or international programmes. Hence, it is seen as important for improving the quality of education and introducing innovations, promoting mobility and cooperation, learning foreign languages, as well as introducing European dimension to the national contexts (e.g. BE(fr), EE, FI, DE, IE, LT and PL among others). More specifically, the Participating Countries argue that Comenius stands out from other programmes by providing the following opportunities: International cooperation between schools would be considerably more fragmented and less extensive without Comenius. Partnerships and projects often prove to be a critical source of the most current international know-how and professional best practice (e.g. see the national report from Estonia). International in-service training of teachers is a unique action, which is not covered by the other existing programmes. As the report from Estonia argues: International in-service training of teachers and educational staff is an invaluable opportunity in a situation where in-service training resources at schools are still scarce and other alternative mechanisms for funding these activities are often absent. Furthermore, the reports argue that implementation of Comenius also complements and creates synergies with the following programmes: Youth in Action programme, as two of them intersect at promoting multilingualism, personal and social development of the participants (e.g. in FR, IE, LT, MT, PL and PT). Nordplus Junior programme is supplemented by Comenius, as well (e.g. Lithuania, Norway). Klavertje Drie (cooperation among schools of three communities of Belgium). Euroklassen (Flemish programme, which promotes cooperation with other secondary schools abroad). European Year of Intercultural Dialogue (2008) (e.g. Ireland). European Year of Creativity and Innovation (2009) (e.g. in Greece and Ireland). BIOS-subsidy (Dutch programme containing pupils and staff s mobility). School partnerships with Africa (in the UK). Interkulturelles Zentrum (Austrian school partnerships). However, it is pointed out that a plethora of similar international opportunities aimed at secondary education introduces some confusion among potential target groups (see the national report from the UK). 14

15 Erasmus. There is a large number of national, bilateral or international programmes in the area of higher education. However, the national reports argue that several features differentiate Erasmus from other existing programmes. First, the geographical scope and accessibility of Erasmus makes it the largest programme supporting mobility of students. As the report from Estonia argues, Erasmus has for long been the main scheme in Estonian higher education providing for such a large scale student mobility, especially on undergraduate level, and staff mobility. Second, Erasmus is a comprehensive programme, which covers a variety of needs faced by the participants in higher education. For example, the report from the UK points out the importance of Erasmus traineeships. The Participating Countries also emphasise that Erasmus has paved a way for a large number of other national or international programmes: Erasmus Mundus is inspired and considerably relied on the networks and instruments created by Erasmus (e.g. the Flemish Community of Belgium, France and Lithuania). Nordplus Higher Education, which is a Nordic version of the original Erasmus programme and hence has major similarities in profile with the LLP Erasmus (the report from Norway). Erasmus Belgica, which, according to the national report from the Flemish Community of Belgium, is a copy of Erasmus but limited to the three communities of Belgium. Other national and international programmes in the area of student mobility usually rely on the instruments created by the Erasmus programme, such as Learning Agreements, Transcript of Records, ECTS etc. (e.g. Estonia, the Flemish Community of Belgium). In addition, the Erasmus programme also complements other national initiatives aimed at personal development of students and lecturers, improving teaching and learning performance, enhancing links between HEIs and industry, encouraging linguistic development, supporting multiculturalism, promoting greater inclusion etc. Furthermore, Ireland and the UK emphasise the contribution of Erasmus to the achievement of the aspiration of the Bologna Process that by % of those graduating in the European Higher Education Area should have had a period of study or training abroad. Leonardo da Vinci. The national reports also argue that the added value of Leonardo da Vinci to other national and international programmes available is high, because it: Provides a unique opportunity for internationalisation (e.g. the Czech Republic, Finland). Widens participation and supports innovations in VET (e.g. the Czech Republic and the UK). Promotes mobility and quality of VET (e.g. Norway, Poland and the UK). The NAs are asked to participate and present their results in all vocational training events organised by the national and/or regional education and training authorities (e.g. in Spain). Leonardo da Vinci creates prerequisites for participation in other projects. For instance, organisations that have successfully participated in projects under the Leonardo da Vinci Mobility action can use their experience to actively participate in new schemes under the Human Resources Development Operational Programme ( ) (e.g. in Bulgaria). Nevertheless, several Participating Countries argue that there are overlaps between Leonardo da Vinci and other schemes. For example the report from the Flemish Community of Belgium points out: The European Social Fund has similar target groups as Leonardo da Vinci and also funds similar training activities. Many stakeholders feel the need for clarifying the relationship and the desired complementarity between Leonardo da Vinci and the training actions supported by the European Social Fund. Such fine tuning and clarification should in the first place be discussed and agreed upon at European level. Austria, Estonia and Poland also support this idea. Grundtvig. In comparison with other education and training sectors, there are comparatively few national or international programmes in the area of adult learning. Hence, the risk of overlaps between Grundtvig and other 15

16 national or international programmes is very low. Instead, the reports argue that Grundtvig complements national efforts aimed at: Stimulating motivation for adult education (e.g. the Czech Republic, Germany and Lithuania). Promoting active international cooperation, development of multicultural projects and mobility of participants in adult education (e.g. the Czech Republic, Ireland and Lithuania). Development and dissemination of innovations in this sector (e.g. Spain). Moreover, this programme complements not only national education policies, but also the economic and welfare policies, as it targets vulnerable social groups (e.g. see the national report from Italy). One of the most striking examples of the added value of Grundtvig is the projects related to education and training in prisons. The national report from the Flemish Community of Belgium indicates that several of the Grundtvig projects with Flemish partners focusing on the education of prisoners are excellent examples how the Flemish Community of Belgium pays attention to the horizontal issues of the LLP. Grundtvig is also seen as highly significant in encouraging socio-cultural advancement of adults (e.g. the French-speaking Community of Belgium, the Netherlands and the UK among others). In addition, Grundvig sub-programme has an increasing role to play in Bulgaria taking into account the challenges related to demographic change, ageing population, literacy and needs for re-qualification. Summary and recommendations To sum-up, the LLP complements and adds value to other existing national and international programmes. Comprehensive package of supported activities, diverse target groups and extensive geographical scope differentials the LLP from other similar programmes. The Participating Countries also provided the following recommendations for further improvement: The Erasmus and Erasmus Mundus programmes should be more integrated (e.g. see the national reports from Austria, the Flemish Community of Belgium, Spain). The Erasmus programme should be more integrated with the Tempus programme, as they both are devoted to education and training (Austria and Spain). Portugal suggests searching for complementarities and synergies with other community programmes, namely Culture Programme, Media Programme, Citizenship Programme and Youth in Action Programme. 2. Effectiveness of the Lifelong Learning Programme 2.1. The level and quality of implementation What is the level and quality of the implementation of the Programme in your country, with specific reference to each sub-programme? What challenges and difficulties do you encounter while implementing the sub-programmes? How do you monitor the implementation of the Programme and sub-programmes? In the view of the Participating Countries, the overall level and quality of implementation of the LLP ranges from good to very good. The level of applications and participants has considerably increased during in the majority of cases. Furthermore, the quality of projects, partnerships and mobility actions has considerably improved. Several factors seem to explain these positive trends: a) better awareness among the target groups of the possibilities and benefits offered by the LLP; b) the participants have developed application-writing and project-management capacities, although this area remains subject to further improvement; c) quality of financed activities is increasingly emphasised in project selection and project management phases. The proactive stance of the NAs has also considerably contributed to these positive developments. 16

17 The national reports univocally argue that participation in the LLP has considerable results and impacts on the individual beneficiaries. Increased language skills, intercultural competences, professional development, higher level of digital competences, enhanced motivation for mobility and international cooperation are among the most often cited individual-level benefits. Furthermore, the majority of the reports also find that the LLP has positive effects on the involved organisations. These include creation of international partnerships and cooperation, changes in the structures (such as establishment of offices for international affairs) and processes of organisations to accommodate mobility of learners. There is also some evidence on changes in the methods and contents of education and training. At the system level the main results of the LLP, as indicated in the national reports, are closely related with the recognition, validation and transferability of acquired qualifications. Despite overall positive assessment, several obstacles hindering more effective implementation of all sectoral programmes were identified. First, reports from AT, BE (nl), FI, IT, PT and SI raise concerns that the main results of the LLP remain circumscribed to the individuals who have directly participated in the funded actions. As the report from Slovenia summarises: <...> there are many activities within the LLP that incorporate high developmental potential, but these are subsequently not reflected upon and developed to the point where they could be potentially interesting for introduction at national level within the education and training system. Consequently, the challenges of harvesting long term LLP impacts remain almost unaddressed. Monitoring visits of higher education institutions for instance showed that institutional outcomes are limited to carrying out administrative and technical activities and fail to find their way into quality assurance mechanisms. Hence, there is a risk that the depth of the impacts at organisational and systemic level could be limited. The second cross-cutting problem refers to the rejection of high quality applications due to limited funds, as indicated in the reports from Estonia, Iceland, Lithuania, the Netherlands, Poland and other countries. This could have negative impact on the level of interest in the LLP, since the rejection of high quality applications could signal low accessibility of the Programme. Thirdly, language barriers remain among the key obstacles to mobility and partnerships. While this obstacle is less relevant for the Erasmus programme, it considerably limits the pool of participants in other sectoral programmes. For instance, the reports from Estonia, Hungary, Ireland, Italy and Turkey note that English language teachers comprise the largest group of participants in the Comenius programme. The Lithuanian and Polish reports show that lack of language skills creates an additional barrier for participation in the Grundtvig programme for the disadvantaged adult learners. This also artificially limits the geography of partnerships: the report from Turkey argues that high levels of partnerships with Germany could be explained by the large pool of Turkish speaking counterparts. Similar reasons explain the high scale of cooperation between Greece and Cyprus. Fourthly, while there is a wide agreement that changes in administrative rules have considerably facilitated management of the projects, the administrative burden is still considered as one of the main challenges to successful implementation of the LLP funded actions. For instance, the report provided by the UK argues: The perception is that administration is burdensome, bureaucratic and rules are inflexible. The administrative costs are considered as particularly high for the financially smaller projects, such as the ones financed under Comenius In-Service Training. The following sections focus on the level and quality of implementation of each sectoral programme. Each section discusses strengths of implementation as well as the relevant obstacles for further improvement. The level and quality of implementation of the Comenius programme. The assessments of the level and quality of implementation of the Comenius programme provided in the national reports range from good to excellent. The number of applications has increased and the quality has improved in the majority of the Participating Countries. The main issues, however, differ by type of action. 17

18 Partnerships are considered as one of the most successful Comenius actions in terms of the number of received applications, satisfaction of the beneficiaries and expected results (see reports from BE (nl), BG, CY, EE, IT, NO, PT and SI to name a few). Flexibility and wide scope of supported activities have clearly contributed to the success of this action. The increasing capacities of schools to develop and manage partnerships have also played an important role, although further progress in this area is necessary. Several factors, however, hinder more active involvement of schools in multilateral and bilateral partnerships. The representatives of the Czech Republic, Estonia and Ireland among others argue that participation in the partnership projects imply prohibitively high costs for some schools. The costs are related with finding and funding substitute teachers and in some cases allocation of own resources for successful implementation of the partnership. In addition, the Austrian and Dutch reports point out to the lack of coordination among NAs in allocating the support for the partnerships. Since all partners need to apply for support in their own country, some partners might receive support, while the applications of other partners are rejected. As a result, such projects are cancelled due to insufficient number of partners. Implementation of In Service Training has been praised due to its positive impact on the beneficiaries. For instance, the survey of trainees conducted in Finland finds that 90% of the beneficiaries have achieved their goals very well or quite well during participation in the action. Other reports also highlight that overall the beneficiaries are satisfied and the action has contributed to increasing linguistic and digital competences, as well as professional development of the trainees. The main barrier to more effective implementation of the In Service Training action refers to uneven quality of the training. The representatives Bulgaria praise the quality of training, while reports from Austria, Finland and the Flemish Community of Belgium point out that in some instances the quality is inadequate. Hence, it is proposed to introduce the minimum quality standards for funded traineeships. Overall, the implementation of Assistantships is regarded as successful. The number of applications from the receiving schools as well as from the potential assistants has increased in the majority of the Participating Countries. Furthermore, the individual beneficiaries are overall satisfied with their participation. Improved linguistic and inter-cultural skills are commonly viewed as the key benefits for the assistants as well as for the students and the staff of the receiving school. Several problems, however, hinder more effective implementation of Assistantships action. First, reports from Austria, Cyprus and Norway among others point towards high rate of cancellations among the selected candidates. This is likely caused by the fact that the recent graduates experience radical changes in their personal and carrier plans, while lengthy selection procedures for Assistantships require longer time commitments. Hence, faster evaluation of applications is likely to reduce the amount of cancellations. The second problem relates to organising assistantships in primary schools. As the report from Norway argues, the majority of applications for hosting assistants come from upper secondary schools, which indirectly indicates that the schools in Norway consider the language skills of their pupils as the decisive factor for using this arrangement. On the other hand, it could be the case that transition from language Assistants in Socrates to teaching Assistants in Comenius is not well understood by the beneficiary schools. The last group of issues relate to country-specific issues. For instance, the limited demand among future teachers in the Czech Republic could be caused by the fact that Assistantships are not always recognised as part of the regular studies. The report from Estonia argues that the motivation to participate is decreased by the fact that graduates lose their social security benefits, <...> if they do not continue studies or get employed. The introduction of Comenius Regio action is welcomed by the vast majority of the Participating Countries. There is a broad consensus that this action has high potential to complement the existing measures. Broadening the target group to include regional authorities and removal of barriers between various sectors of education 18

19 these are the largest benefits of Regio, as argued in reports from Austria, Bulgaria, the Czech Republic, Hungary and Italy. It seems that Regio action has been particularly successful in Austria, Italy and the UK. For instance, the report from the UK argues that the programme has proved very popular in the UK, and one third of all 2009 Regio partnerships across Europe involve a UK partner. On the other hand, CY, EE, IE, IS, LT, LU and NO have faced problems in securing sufficient number of applications. This could be explained by the fact that the Regio action is rather new. Hence, difficulties in energising regional authorities and municipalities could be attributed to the lack of experience and social infrastructure for dissemination of information about the action to the potential beneficiaries. Furthermore, the report from the Flemish Community of Belgium draws attention to the fact that the programme design does not suit all countries equally well. For instance, the new Comenius Regio projects are geared towards the communes and the provinces. From a Flemish perspective it would have been more effective if the education umbrella organisations were directly involved. Despite the above discussed problems faced in the initial phase, there is considerable optimism regarding the potential usefulness of the action in the future. The level and quality of implementation of the Leonardo da Vinci programme. Overall, the Participating Countries argue that the level and quality of implementation of the Leonardo da Vinci programme is good or very good. The beneficiaries are satisfied with their participation in the programme (although assessments provided in the report from Spain are less favourable). Furthermore, the potential participants show an increasing interest in the programme (especially in Iceland there is an increasing demand due to the economic crisis). The success of Leonardo da Vinci could be illustrated by the fact that the Austrian government decided to provide additional funds from the national budget with the view of increasing the scale of the programme. Mobility actions are considered as especially successful in AT, BE (nl), BE (fr), CY, HU, IS, LT and NO. This is largely related to the direct impact of mobility actions on the learners. The surveys of beneficiaries (carried out in BE (fr), EE, FI, IE, LT and PL) found that mobility has contributed to increased language skills, intercultural competences and professional development. Furthermore, there is some evidence that placements have directly contributed to enhanced employability of the trainees (see the reports from Bulgaria and Estonia). Partnerships and innovation transfer projects are seen as an important instrument for exchanges of the best practice. Focus on the development of concrete products or outputs is considered to be one of the most important preconditions for the success of such projects, as argued in the reports from Estonia and the Frenchspeaking Community of Belgium. However, there are several obstacles that hinder more effective implementation of Leonardo da Vinci: Lack of language skills prevents higher levels of mobility of learners and trainers. This issue is particularly emphasised in the reports from the Flemish Community of Belgium, the French-speaking Community of Belgium, Poland, Spain and Turkey. Difficulties in developing partnerships (with schools, but especially the host employer and SMEs in particular) are pointed out in the reports from BE (nl), BE (fr), ES, FR, IT, NO and TR. Austria and the French-speaking Community of Belgium among other countries have encountered difficulties in securing adequate level of participation of the trainers in mobility actions. A large number of countries emphasises the benefits of simplification of administrative and financial procedures and introduction of lump-sum transfers in particular. For instance, the report from Hungary argues that this allowed shifting focus on the content and quality of the Leonardo da Vinci actions. Nevertheless, the report from Norway points out that further progress in this area is necessary, as the administrative costs for preparation of an application and management of the project tend to be prohibitively high and in some cases exceed the benefits of the grant. The level and quality of implementation of the Erasmus programme. The Erasmus programme remains a flagship of internationalisation of higher education. Overall, the national reports argue that implementation of this sectoral programme is very successful. This is largely connected with the impacts of the programme on 19

20 individual beneficiaries and higher education institutions. Students mobility considerably contributes to language learning, understanding of the diversity of cultures and personal development. Teachers and other educational staff s mobility is seen as important for the professional development of the beneficiaries and has a positive impact on the students motivation in the recipient higher education institutions (HEI). In addition to individual-level benefits, mobility actions have also considerably contributed to internationalisation of HEIs. For instance, attempts to facilitate students mobility led to establishment of offices for international affairs, which expanded their functions to encompass direct contacts with other HEIs. Furthermore, development of intensive programmes, participation in networks and multilateral projects has contributed to higher intensity of cooperation between HEIs. As the national reports from Austria and the French-speaking Community of Belgium argue, such collaboration has culminated in the development of joint degrees. In the majority of the Participating Countries, the number of outgoing students has increased during As the reports from the Czech Republic and the Flemish Community of Belgium argue, introduction of such instruments as ECTS, Learning Agreements and Transcripts of Records has removed important barriers for student mobility. Low administrative burdens for students and high accessibility have also contributed to the success of Erasmus. Despite the overall success and unanimous support for the student mobility action, the national reports draw attention to several implementation problems. First, the numbers of outgoing students have not been rising as fast as expected (e.g. in Italy) or have been decreasing (e.g. in Norway). The report from Italy argues that this could be partially explained by the fact that Erasmus study mobility has now reached almost two thirds of the eligible higher education institutions, approaching the saturation of the catchment area. Hence, it is reasonable to expect that the rate of growth of outgoing students should slow down in the countries that have been the most successful in this respect. Second, there is a nearly unanimous agreement that inadequately small grants for mobile students (and teachers) hinder more effective implementation of Erasmus. On the one hand, this poses an obstacle to further increase in the numbers of mobile students. On the other hand, limited size of the grants has negative effect on equal opportunities: students from less well-off families face disincentives to participate in the programme. The level and quality of implementation of other decentralised Erasmus actions overall is good, although progress differs. The intensive programmes are very popular in Austria and the Flemish Community of Belgium, although reports from Cyprus and the French-speaking Community of Belgium point out to slow improvements in this area. Similarly, student placements have seen sharp increases in the number of participants in Austria and Italy, while strengthening the links between universities and enterprises remains an area for improvement in the Czech Republic, Estonia and Lithuania among other countries. The level and quality of implementation of the Grundtvig programme. The national reports argue that the level and quality of implementation of the Grundtvig programme is good or very good. This is indicated by several factors. First, the number of applications (particularly for mobility actions) has increased in a majority of the Participating Countries. Second, as the reports from Bulgaria, the Czech Republic, France, the French-speaking Community of Belgium and other argue, the programme has become increasingly accessible for a range of target groups, including those with special needs, immigrants and prisoners. Third, the learning partnerships have resulted in particularly high quality outputs in BE (nl), BE (fr), CZ, CY, ES, FI, LT, NO and other Participating Countries. Furthermore, In-Service training has demonstrated considerable potential for enhancing the competences of the beneficiaries and increasing the overall quality of adult education (e.g. see the reports from Finland and Ireland). On the other hand, several obstacles are encountered during implementation of the Grundtvig programme. The first one is closely related with the diversity of adult education sector. The learning opportunities are provided by a diverse range of actors. As a result, implementation of Grundtvig faces difficulties in involving the diversity 20

21 of actors in the programme. For instance, the report from Poland argues that lack of umbrella associations and institutionalised system of adult learning hinders successful targeting of potential beneficiaries. The representatives of Cyprus argue that teaching staff from public adult education centres comprises the majority of beneficiaries, while more active involvement of private and non-governmental organisations poses significant problems. Similar challenges are also faced in Spain. However, the opposite is true for the French-speaking Community of Belgium, where the majority of beneficiaries represent non-governmental sector. A different problem is encountered by the beneficiaries from Turkey: problems in obtaining visa are outlined as one of the most important obstacles to successful implementation of Grundtvig (and other actions). While simplification of administrative requirements has been praised by several countries, other Participating Countries argue that further progress is necessary in this area. The report from Austria argues that high administrative workload creates disincentives for smaller organisations to participate in the programme due to the lack of administrative resources. Furthermore, Bulgarian and Lithuanian reports emphasise that smaller organisations face financial constraints of participation, since project managers are reimbursed only after submission of the final reports. Similarly, the Hungarian National Authority points out that implementation of a number of good project-ideas have been obstructed by administrative hurdles. Introduction of four additional types of actions in 2009 caused some additional problems. As the report from the UK argues, The number of different actions, different rules and different application deadlines and occasional uncertainty about the interpretation of the rules, made it difficult to clearly present the programme to interested stakeholders and has increased the management time needed to run it. The report from Spain also presents similar concerns: the limits regarding eligible beneficiaries and the respective objectives of some mobility actions are not clear and difficult to pinpoint, which generates certain confusion among potential users. In addition, France also argues that a number of different actions is excessive. The level and quality of implementation of the study visits. Overall, the level and quality of implementation of the study visits under the Transversal programme is considered as successful. Several countries (including Italy, Poland and Slovakia) however, struggle with the involvement of key decision makers. Furthermore, the reports from Estonia and Turkey argue that in the view of the target groups, the study visits fall-out of the programme, since the name-recognition of the Transversal programme is rather low. What are the instruments for monitoring the implementation of the Programme? The Participating Countries have established an extensive multi-levelled and multi-layered monitoring system. Virtually, every state uses the following monitoring instruments: consultations, monitoring seminars (consultative, thematic, group monitoring seminars), monitoring visits, on-the-spot checks, assessments of final and interim reports (including subsequent feedback), comprehensive check of financial documents, daily contacts via and Telephone with the beneficiaries. The national reports argue that thematic monitoring seminars are particularly effective: in addition to generating qualitative insights into the success of the projects, this instrument also facilitates networking, exchanges of good practice, peer-learning and valorisation of key outputs. Several countries have also developed new monitoring instruments. Estonia has developed web-based surveys of beneficiaries. Italy uses similar surveys to track the impact of the Leonardo da Vinci programme on the beneficiaries. Furthermore, Italy has also adapted the M-tool to organise online thematic monitoring forums. Furthermore, simplification of financial monitoring procedures is praised in the reports from Hungary and Norway. This has allowed shifting more energy and resources: a) from control and supervision towards advice and support; b) from financial checks towards monitoring of quality of the activities. Several issues, however, prevented more effective use of monitoring instruments: Frequent changes in monitoring procedures and instruments. 21

22 High costs of monitoring activities. As the reports from Austria, Finland and Norway among others argue, it is doubtful whether the costs of monitoring are justified by the benefits. The report from the Netherlands argues that the monitoring system is ill-equipped to draw solid conclusions of the programme: There is no rationale behind the targets set, other than a calculation of the available funds and the results of previous years. The targets are not related to objectives of the programme. Summary and recommendations Overall the assessments of the level and quality of implementation of the LLP range from good or very good. Nevertheless, the national reports provided a number of recommendations for further improvements in this area. The cross-cutting recommendations pertinent to implementation of all sectoral programmes are the following: The management of the LLP should be simplified (e.g. CZ, DE, EL, ES, FR, IS, LT, NO, PL, UK). Poland, Spain and the UK point out that simplification of procedures (e.g. process automation based on development of responsive and adequate ICT tools, simplified administrative and financial rules) will contribute to the higher quality of the LLP management. Lithuania among other argues that uniform management procedures should be introduced for similar activities. Further capacity building is necessary. Cyprus, Ireland, Luxembourg, Slovakia and the UK among others argue that training of the NAs staff deserves more attention. Skills necessary for working with the management supporting tools represent the largest training gap. There is also a need to continue building application writing and project management capacities of the potential beneficiaries of the programme (e.g. see reports from Cyprus, Poland, Slovakia and Turkey). The Programme should be stabilised and changes should only be introduced for the purpose of making the implementation or management process easier and less complicated (see the reports from CZ, DK, EE, FI, HU, LU and PT). Stable rules will considerably contribute to higher quality and efficiency of management and credibility of the LLP. Higher stability is particularly important in such areas as information system, the project forms, the application forms and the financing rules. The introduction of lump-sums should be further encouraged (see the national reports from Finland, Lithuania and the UK). Further reduction of administrative costs of application should be encouraged. For instance, reports from France and Finland recommend that successful implementation of projects, mobilities or partnerships should lead to certification or accreditation. This could be used to facilitate further involvement in the LLP actions. Further rationalise the structure of the LLP. The report from the UK suggest: To enable learners and learning professionals to access the same types of Mobility actions across the Programmes. This will simplify the range of options available and ensure that participants are not excluded from the Lifelong Learning Programme because they do not fit the criteria of a particular programme. Further emphasis on promoting the links between education and labour market is needed. The report from Greece suggests that all sub-programmes could be aligned more explicitly to the EU Key Competences Framework and horizontally meet the objective of a multi-faceted participant re-skilling. Furthermore, there is a also a need for concerted action aimed at more intensive involvement of enterprises in the implementation of the programme, since finding partners for traineeships remain an issue in a large number of countries. Specific recommendations for further improvement of Comenius programme are as follows: Increase the flexibility of the Programme. Currently the requirements of Comenius are seen as too rigid. Higher flexibility is particularly emphasised in the three areas. First, report from Slovakia suggests that the funding rules of individual mobility actions should be relaxed: it should be easier to move reallocate funds between the budget lines of the funds. Secondly, the French-speaking Community of Belgium and 22

23 Lichtenstein argue that regulation of the duration of projects and partnerships should be more flexible, in order to account for the design and objectives of the different activities. Promote mobility of pupils. The Participating Countries (including Austria, the Flemish Community of Belgium, France and Iceland) suggest that more resources should be allocated to this type of action, as it is seen as particularly relevant and effective. The quality of mobility could be ensured through the implementation of institutional agreements (similar to the ones used in Erasmus). Reduce the costs of participation in the Comenius Programme. On the one hand, schools face high administrative burden during the application process. These costs could be reduced by introducing flexible longer term framework contracts for schools. On the other hand, the sending schools of in-service trainees as well as the trainees themselves face high costs of participation in the Programme. Hence, Denmark recommends providing financial incentives as well as tools for recognition of acquired qualifications (certificates, etc.) to motivate participation in Comenius. Reduce the risks of participation in Comenius partnerships. Currently each partner needs to secure funding at the national level. This implies a risk, that not all potential partners might be able to secure funding and implement planned activities. Hence, the Austrian authorities recommend that the entire partnership should be funded by the participating country, where the coordinating partner is established. Exploit the potential of the etwinning. As the report from Finland argues: To complement the physical mobility actions, virtual mobility should be increased at educational levels. This would be useful especially in school level, where the large target groups would have an equal, sustainable as well as cost-effective possibility to participate in the mobility actions. In addition the UK report recommends that local authorities should be also granted access to the etwinning. More attention to language learning, because it remains one of the key obstacles to enhanced participation in Comenius. The Flemish Community of Belgium recommends that Comenius should provide an opportunity to teachers to participate in European Intensive Language Courses. Further strengthen Comenius Assistantships. Report from the Flemish Community of Belgium argues that this could be done by splitting up up the Comenius assistantships into two strands: one for students once they are graduated and one for students during their initial teacher training linked to Erasmus. Furthermore, the Slovak Authorities urge to secure adequate quality of this action: there is a need for a clear agreement on the assistant rights and obligations stipulated in a contract, explicitly pointing out that using an assistant as a deputy teacher as a substitute of missing teachers is inadmissible. Involve regional and local authorities in implementation of the Comenius. Reports from Austria and the UK praise the benefits of Comenius Regio and suggest that there is a need for more extensive and flexible involvement of local authorities in the implementation of the Comenius programme. Specific recommendations for further improvement of Leonardo da Vinci programme are as follows: Reduce the administrative burden of participation in Leonardo da Vinci. Reports from the Flemish Community of Belgium, the French-speaking Community of Belgium and Slovakia argue that: different parts of the project applications require the same information, which leads to unnecessary duplication of work; the structure of applications for mobility actions and evaluation sheets are not entirely compatible; monitoring puts too much emphasis on processes and too little on outputs and outcomes. Increase flexibility of the programme. The report from the UK argues that introduction of shorter mobilities (for e.g. 1 week) would enable participants who are in full time employment to benefit from a Leonardo work placement. This would widen participation from participants in SMEs and disadvantaged groups who find it difficult to undertake a 2 week placement. Furthermore, report from the French-speaking Community of Belgium recommends that the duration of the Transfer of Innovations projects should be more flexible. Continue work on the recognition of qualifications. Recognition of non-formal qualifications remain a major challenge for further European collaboration in the area of VET. Therefore reports from Austria and the UK recommend that the next generation of the LLP should allocate more resources towards tackling this issue. Facilitate emergence of networks. Finding partners for Transfer of Innovation and mobility actions remain a challenge. Therefore, the report from Slovakia recommends organising pan-european contact seminars, while report from Spain suggests that the European Commission should provide support to the networks of actors focusing on VET. Put more efforts towards involving enterprises in the Programme. This remains a challenge for a large number of countries. Hence, France suggests providing financial incentives to companies that encourage their workers to participate in mobility actions or are willing to take-up trainees. 23

24 Specific recommendations for further improvement of Erasmus programme are as follows: Increase the flexibility of the programme. The past few decades witnessed increases in the number of parttime students, who combine education with work or family responsibilities. However, Erasmus mobility actions are ill-suited to cater for the needs of these students, who often cannot spend a full semester abroad. Hence, Austrian, Greek and the UK Authorities suggest that there should be a possibility for shorter student mobility periods (for e.g. 2 3 weeks). The report from Slovakia also proposes to increase flexibility of the non-academic personnel mobilities: Good organisation and preparation may result in shorter effective stays, in particular where the travelling time is short. It is proposed to consider shortening of the minimal duration to three days of stay. Facilitate access to the programme. Since the level of financial support provided to the mobile students is barely adequate to cover the differences in living costs, students from less well-off families face disincentives to participate in the programme. Hence, it is recommended to establish a segregated fund to support students, who cannot co-finance their mobilities. Provision of addition support could be meanstested. Balance geographical scope and subject coverage of student mobilities. The national report from Greece recommends providing support for students who wish to visit countries of less commonly spoken and taught languages, including linguistic and cultural preparation. Reinforce the emphasis on the links between higher education and the labour market. Involving employers in Erasmus programme and locating companies willing to take on students for internships remains a challenge in a large number of Participating Countries. Hence, the report from Spain recommends using the databases of the Eures network in order to disseminate the programme among European companies, as well as to offer training placements through the portal. Specific recommendations for further improvement of Grundtvig programme are as follows: Streamline the programme. There is a nearly universal consensus that Grundtvig has too few resources and too many distinct target groups and types of actions. This diminishes the potential for considerable impacts and creation of a critical mass. However, the national reports do not agree on concrete measures that would allow streamlining the Programme. The Flemish Community of Belgium suggests creating an opportunity for focusing on particular target groups, sectors or themes. Norway recommends adoption of fewer and more general types of activities. Slovakia explicitly supports the idea of prioritising the participation of providers of adult education. The UK argues that instead of introducing new types of actions, better horizontal integration is necessary: for instance, Senior Volunteering could be better linked to the Commission efforts related to active citizenship. Increase the flexibility of the programme. Report from Romania advocates extension of the duration of study visits. Polish Authorities recommend that duration of the mobilities during Senior Voluntary Projects could be reduced to 2 weeks and the duration of projects could be reduced to 1 year. Reports from Slovakia and the UK suggest considering a more flexible system for funding individual mobilities: the regulations should allow transfer of funds between budget categories. Furthermore, the Flemish Community of Belgium recommends creation of etwinning mechanisms for adult education and training: this should facilitate involvement of adult part-time learners. Facilitate language learning. Report from Slovakia suggests that language learning (particularly of less used and taught languages) should be introduced as new type of activity. However, the UK Authorities suggest that in order to contain expansion of the types of activities, language learning should be horizontally integrated into all other activities. Rationalise the management of the programme. As the report from the UK argues: Within Grundtvig, which has just 4% of the budget, there are 7 different actions. This means that funding is spread thinly and there is a lot of work running so many different actions. There is a strong case for rationalisation, for example there are three actions for professional staff development (IST, V&E and Assistantships) each of which has a different applications form, different application timetable, etc. We d propose that this is simplified and that an adult educator can apply for an activity from a menu of short courses, conferences, work shadowing or longer placements. Specific recommendations regarding the instruments for monitoring the implementation of the Programme: Reduce the administrative workload related to monitoring and supervision. There is a nearly unanimous agreement behind this recommendation. Reduction in the workload could be achieved through further standardisation of electronic monitoring and reporting forms, assessment of duplications of requested information, timely introduction of electronic monitoring and management-support tools as well as 24

25 application and reporting forms, introduction of uniform funding and management procedures for similar activities, etc. Further work is needed in setting targets and indicators. This is particularly important for systemic measurement of the impacts of the LLP. Monitoring requirements should be stable, unless the changes reduce the management workload Integration of the LLP What are the advantages and disadvantages of the integration of previous programmes into one LLP for your country? What are the effects of the integration? Can you identify an added-value produced by the integration within the LLP in your country/in general? In what respect? How do you foster the synergies between the different parts of the LLP? How do the centralised and decentralised actions complement each other? Which are the most effective sub-programmes and actions considering the national needs in the field of education & training? What negative and positive factors seem to be influencing outputs and results? Did the integration of the previous programmes have an effect on the groups targeted by the Programme? Have new target groups joined the LLP? The National Authorities argue that the integration of previous programmes into a single Lifelong Learning Programme is a positive initiative. The current structure is seen as a logical attempt to develop a programme that covers all stages of education from cradle to grave. The main benefits of integration have occurred in two main areas: a) management of the programme and b) dissemination of information to the target groups. Nevertheless, the majority of the reports argue that the synergies between the sectoral programmes as well as between the decentralised and centralised actions remain underexploited. Advantages and disadvantages of the integration from administrative perspective. There is a nearly unanimous agreement that integration has considerably facilitated administration of the Programme. Key benefits have occurred in the following areas: Standardisation of procedures, practices and documents has contributed to more efficient management. More specifically, efficiency is enhanced by the development of uniform IT tools; standardisation of procedures and documents used for the applications, reporting, monitoring and supervision purposes; development of uniform procedural and administrative rules for the staff of the Agencies. Integration has resulted in more efficient promotion and information campaigns. Higher efficiency results from the following: the development of a single website (instead of several websites for each programme), a single design of all dissemination material and organisation of promotion events that cover more than one sectoral programme. Development of a unified database for all supported projects has contributed to more effective control of fraud and possible double funding. Integration has widened the opportunities to re-allocate funds across the programmes. Since participation in different actions fluctuates, these opportunities have facilitated timely and flexible decision-making in this respect. In some countries (e.g. in Estonia, the French-speaking Community of Belgium and Iceland) integration of the programme has resulted in the merger of several organisations, which previously managed Erasmus, Leonardo da Vinci and Socrates on a separate basis. This contributed to higher level of coordination and reduced duplication of similar administrative tasks, which were previously carried out in different Agencies. The national report from the French-speaking Community of Belgium points out that the administration of the finances is less complicated after the integration of the programmes. 25

26 The Finish Authorities argue that after the integration, the LLP has gained more visibility and prestige at the policy level. The national reports from Germany and Sweden note that the administrative procedures have become more transparent. From the administrative perspective, however, the unification of procedures also had several negative sideeffects. First, unification and standardisation in some instances has gone further than actual integration of programmes and activities. As the report from Hungary argues, <...> exaggeration in bundling of programs carries a clear threat and approximation sometimes has no function (e.g. the attendance of common meetings may be too heterogeneous, and therefore no common "message" can be created). Forced standardisation may lead to losing specific differences that should be preserved. Standardised forms are mentioned as an example along with standard reports that leave no room to cover certain nation-specific aspects (e.g. types of institutions) or sector specific solutions. Second, the reports from the French-speaking Community of Belgium, Lithuania, Luxembourg and Spain argue that integration coincided with an increase in administrative workload, which resulted from introduction of additional obligatory rules and procedures. Thirdly, France emphasises the fact that despite the integration, the procedures for different programmes vary considerably. Lastly, the report from Lithuania points out that integration at the national level was deeper than administrative integration within the DG EAC. The report argues that there is weak LLP integration at the Commission level, no coordination of activities of its units and similar activities under different sub-programmes are managed differently <...>. Did the integration of the previous programmes have an effect on the groups targeted by the Programme? There is a divergence in the opinions regarding the effects of integration on the groups targeted by the Programme. On the one hand, reports from Estonia, the Flemish Community of Belgium, Norway, Spain and Portugal argue that integration resulted in better accessibility of information, because: The NAs play the role of a single information point on all aspects of the Programme. The structure of the Programme is more transparent. This, as the report from the UK argues, makes it easier for organisations to discover the different programmes and the ways in which they can help meet the needs of the different target groups they serve. Furthermore, the reports from BE (fr), CZ, EE, LT, NO and PL argue that integration has contributed to higher awareness of the possibilities offered by the sectoral programmes. This is particularly relevant in the case of transfer of student placements from Leonardo da Vinci to Erasmus. Previously the students were either unaware of placement opportunities offered by the Leonardo da Vinci, or considered that they are not eligible to participate in the Leonardo da Vinci actions, which were associated with vocational training. The transfer of action to the Erasmus programme contributed to higher interest in student placements due to better visibility of this action among this target group. On the other hand, reports from BE (nl), BG, CZ, DE, DK, FI and SK argue that integration, from the perspective of potential beneficiaries, has not made a considerable effect. As the Danish report points out, Most project managers (based on the qualitative interviews) say that they have not noticed the merger and/or that they have not felt a difference at all. <...> Only a few project managers have noticed a change. Furthermore, the report from Bulgaria argues that attraction of new target groups does not directly depend on the integration of the programmes. It is rather related to their increased popularity and an awareness of their usefulness. New target groups (municipalities and Regional Inspectorates of Education) have been attracted as a result of launching new activities, as for example the Comenius Regio partnerships. What are the synergies between different parts of the LLP? The reports point out to three main areas of synergies between the sectoral programmes: Austria and Ireland among other countries note the synergies between etwinning and other Comenius actions. As the Irish report argues, Synergies between etwinning and Comenius continue to enhance 26

27 autonomous learning at all levels. Using ICT, two or more schools from two or more European countries can work in partnership on a project of their choosing. This becomes a valuable learning experience in which pupils and teachers alike encounter new cultures while developing ICT skills. The etwinning programme provides tools for carrying out the project. The mailbox allows pupils and teachers to exchange messages safely, while chatrooms make live-feed communication possible. The flexible nature of etwinning means that projects can last as long as the partnership decides and there is no restriction on themes. Integration into the curriculum is recommended but often this happens quite seamlessly, as the pupils discover the myriad possibilities of the technologies available. There are potential synergies between the study visits funded by the Transversal programme and the sectoral programmes. The study visits have the potential to result in joint projects and partnerships funded by the sectoral programmes. There are also synergies within the programmes. As the report from the Flemish Community of Belgium argues, for instance, student and teacher mobility often leads to the setting up of Intensive Programmes. These in turn might lead to Curriculum Development (CD) projects or Joint Degrees. Thus, centralised and decentralised actions complement each other. In addition, the French-speaking Community of Belgium points out that the synergy at the level of evaluation was created. For instance, the same deadlines for the applications have been set. As Austria summarises, the synergy is clearly visible at the administrative level, as the unification of administrative procedures (e.g. various forms, requests and deadlines) increases the accessibility of the LLP as well as the efficiency of its management. Nevertheless, several Participating Countries (including Denmark, the Flemish Community of Belgium, the French-speaking Community of Belgium, Hungary and Slovenia) strongly argue that the synergies between different parts of the LLP are surprisingly limited. This is largely associated with segmentation of the Programme. As the report from the Flemish Community of Belgium argues, Actually, the integration of the programmes into the LLP is perceived to be more an addition or juxtaposition than a real integration. There was almost no streamlining of actions: the LLP has no less than 64 actions and sub-actions! Moreover, there are still many walls between the sub-programmes <...>. Accordingly, these countries would favour further integration of the LLP. How do the centralised and decentralised actions complement each other? The national reports agree that there is a considerable scope for complementarities between the centralised and decentralised actions. Nevertheless, the potential for complementarities remains underdeveloped due to lack of collaboration between the agencies that manage centralised and decentralised actions. Furthermore, a vast majority of the Participating Countries argue that they cannot assess the complementarities due to lack of knowledge about the centralised actions. Which are the most effective sub-programmes and actions considering the national needs in the field of education & training? Overall, the Participating Countries argue that all sectoral programmes are seen as very important within the respective national systems. Summary and recommendations To sum-up, the Participating Countries welcome the integration of the LLP. However, the potential for synergies between the sectoral programmes and between centralised and decentralised actions remain limited. The main areas that require further improvements include: Increasing the synergies between the programmes as well as between the centralised and decentralised actions. Denmark points out that synergies should be ensured not only internally in the NAs but also towards the beneficiaries, who should be supported and inspired to find ways of combining activities also across to the Youth in Action Programme or centralised measures, where applicable. Poland argues that < > it is particularly important not only to conduct promotional activities, but also to support exchange of experiences and opinions between the programmes, exploit good practices obtained in other programmes 27

28 by unifying management support tools (taking also into account specificity of sectoral sub-programmes). The UK summarises that a more integrated approach between centralised and decentralised projects would encourage more synergies between centralised and decentralised programmes especially between Leonardo da Vinci Transfer of Innovation and Development of Innovation. Ensure better flow of information between the NAs and the EACEA. This could be done during regular meetings, through a dedicated websites and other information sharing instruments. Further standardisation of processes, procedures and documents with the view of increasing efficiency and reducing administrative workload. The standardisation, however, should take due account of the differences in the types of actions and target groups: the one size fits all approach should not be overemphasised. Further decentralisation of decision making should be considered (see reports from the Czech Republic, Germany, Hungary, Lithuania and the Netherlands). It is expected that decentralisation should lead to better adaptation of the programme to national contexts. Finland notes that more implementation issues should be transferred to the NAs, while the Committee could concentrate on more political issues. As a result, the most important projects and specific target groups can be better supported. However, the attitudes towards decentralised management should not be unduly universalised, since they should flexibly adjust to the national needs Reaching the target groups How successful is your country in reaching the sub-programme's target groups? All the Participating Countries argue that overall the Programme successfully reaches its target groups. Increasing quality and number of applications for a majority of actions is a good indicator in this respect. Nevertheless, several general as well as programme-specific problems remain. First, BE (nl), BG, DK, EL, ES, RO and SI argue that despite considerable efforts, persons with special needs remain the most-difficult-to-reach target group. For instance, France points out that schools that are educating pupils with special needs barely participate in the Comenius actions. Several factors could explain low participation of this target group. The report from Romania argues that the main barriers are related to the lack of persons with special needs mobility tradition. Report from Spain indicates that larger grants for this target group could be beneficial. The Flemish Community of Belgium, in contrast, argues that the size of the grant is not the main problem: more active participation of persons with special needs is prevented by the uncertainty regarding the facilities and the possible absence of carers abroad. The second cross-cutting issue relates to the balance between old and new applicants. As the report from the Netherlands argues, the promotion activities might reach new groups of potential beneficiaries, but new applicants might have less chance to be selected, due to limited experience with drawing up applications. <...> This process has a negative and a positive effect. It is positive to see that networks and partnerships have the opportunity to build on previous experiences and further increase their cooperation. The negative effect is that the LLP in some cases becomes more a structural subsidy and loses its stimulating function for new players. The national report from Sweden also emphasises the problem of reaching new target groups, as they lack the knowledge about the participation opportunities in the LLP. Thirdly, inadequate (although improving) geographical coverage of the applicants remains an issue FR, LT, PL, TR and to some extent BE (fr) and EE. It seems that this problem is not directly related with dissemination of information about the opportunities offered by the LLP. The report from Norway supports this view: once a school in one area has had a project, more schools in the same area apply the following year, so dissemination and recruitment functions well at the local level. Instead, lack of project development and management skills as 28

29 well as inadequate language competences constitute the most important barriers for participation of beneficiaries from rural areas, as pointed out in the Lithuanian report. In addition to the above discussed cross-cutting issues, each sectoral programme also faces problems in reaching specific target groups. Attempts to reach the target groups of Comenius have faced the following problems: Lack of language skills remains an important barrier to participation of all target groups in the Comenius programme. As a result, English language teachers comprise the largest group of participants in the Comenius programme, as indicated in the reports from Estonia, Hungary, Ireland, Italy and Turkey. Involvement of target groups from primary and pre-primary education sector remains problematic in the Czech Republic, the French-speaking Community of Belgium, Lithuania and Norway. This, however, is not the case in Iceland, where over 60% of all primary schools have already participated in Comenius. Implementation of Leonardo da Vinci in some countries faces challenges in reaching the following target groups: More active participation of enterprises (SMEs in particular) is emphasised by BE (fr), DE, DK, EE, HU, LT, SI and SE). Problems in targeting female students are pointed in the report from Turkey. As the report argues, the target group of this programme in Turkey (students of vocational high schools, apprentice training centres and business environment) already consists mainly of men compared to women. Furthermore, women in this target group are less reluctant to apply for the Leonardo da Vinci programme compared to men, because of their social responsibilities, such as the burden of domestic labour and their families. The report from Sweden points out that it is difficult for the unemployed persons to participate in Leonardo da Vinci, since the beneficiaries from vocational schools are still dominating. Similarly to Leonardo da Vinci, involvement of enterprises remains an important issue in implementation of the Erasmus programme. Furthermore, a report from the Flemish Community of Belgium argues that Erasmus is less successful in reaching the (adult) students, who combine work and study. As the report argues, very often those students cannot be involved in Erasmus mobility for practical reasons; they are also excluded from Leonardo da Vinci and Grundtvig mobility: for Leonardo da Vinci they are explicitly excluded and in Grundtvig preference is given to adults who are no longer in higher education. Moreover, France points out that social sciences, business and law students are much better represented than those studying natural sciences. However, this issue found a solution in Luxembourg: all students from the University of Luxembourg are obliged to study at least one semester abroad regardless their studying subject. The Grundtvig programme targets a large number of diverse groups. They range from unemployed seniors to refugees, from prisoners to senior volunteers. Hence, the limited budget of the programme implies that only a few people from the distinct target groups could be reached. The problems related to the diversity of potential beneficiaries and creating a critical mass are particularly emphasised in the reports from BE (nl), ES, FR, LU, SI, TR and UK. Germany points out that the adult education institutions are much better represented than cultural organisations or NGOs, which are hardly affected by the programme. Furthermore, the report from Poland argues that inadequate language competences remain an important barrier for the target groups participation in Grundtvig. In addition, implementation of the programme in the Czech Republic faces the challenge of involving seniors in Volunteering projects. However, the national report from Austria argues that various target groups, including migrants and persons with low educational level, are successfully reached. Such a result was achieved due to specific promotional events. Besides a high number of country reports (CZ, GR, NL, NO, SP) indicate that the programme is only known to a very specific target group. Summary and recommendations Overall, the majority of the target groups are successfully reached. Nevertheless, there is considerable scope for improvements in the following areas: 29

30 More efforts should be put in reaching the end-users, who have not yet participated in the Programme. Report from Norway recommends strengthening the role and contributions of the social partners to meet this challenge. Furthermore, there are also considerable cross-national variations in the extent to which target groups are prioritised. For instance, Denmark refers to the single mothers and persons with disabilities, while Turkey emphasises the need to improve accessibility to the learning opportunities for the women overall. Geographically distant areas (located at the peripheries of the cities) should also be taken into account (e.g. BE (fr), EE, FR, LT, PL and TR). In order to reduce inadequate geographical coverage, Slovakia suggests organising seminars for managerial staff in schooling and potential project managers in those peripheral regions. Bulgaria proposes to focus on early school leavers and persons without basic skills. The Czech Republic notes: As for the future programme, the Czech Republic would like to boost and underline the mobility of teachers and school leaders. This target groups is of particular importance not only in terms of the management and quality of education, but also in terms of dissemination of the outcomes and knowledge obtained. These individuals are key multipliers of the mobility impact. The UK suggests that the most disadvantaged groups could get prioritised funding from the NAs at the national level. Furthermore, specific cross-cutting initiatives for people with disabilities could be developed. Alternatively, there should be European priorities to ensure that funding can be targeted at key groups. There is a need to promote partnerships between business world and educational institutions (see the reports from the Czech Republic, Greece, Italy and the UK). As Denmark points out, the involvement of companies in projects is an area where the evaluators see that more synergy and more relevance could be built up. Spain suggests using the databases of the Eures network in order to disseminate the LLP among European companies, as well as to offer training placements through the portal Contribution to the EU priorities To what extent is the LLP implementation at national level contributing to the EU priorities as set out in the Education & Training Programme 2010? All national reports implicitly or explicitly argue that implementation of the LLP at the national level contributes to the priorities established in ET As the report from Estonia puts it, the Programme was by default set up so that it would contribute to these priorities. The most direct links between the LLP and priorities of the ET 2010 are found in the following areas: The LLP actions such as In-Service training, Assistantships, Professionals in vocational education and training, partnerships and projects aimed at developing new methods and approaches, directly contribute to the ET 2010 objective of Improving education and training for teachers and trainers. Mobility of learners and providers of learning opportunities has contributed to the ET 2010 objective of Developing skills for the knowledge society. A considerable proportion of projects and networks aim to develop new ICT based content, pedagogies and practices and/or use ICT in implementation of the activities, which directly or indirectly contribute to Ensuring access to ICT for everyone. In some countries (e.g. Bulgaria) implementation of the LLP is linked with modernisation of the educational system, which is associated with Making best use of resources. The ET 2010 objectives aimed at Improving foreign language learning cut across all actions of the programme. Every programme includes mobility actions, which are directly linked with the objective of Increasing mobility and exchange. Implementation of mobility actions as well as projects and partnerships directly contributes to Strengthening the European cooperation. For example, the report from Portugal argues that at the organisational level, the most recognised results are those that stem from the cooperative relations developed. 30

31 Facilitating the access to the education and training systems, for example, through integrating illiterate people, prisoners, visually impaired and hearing impaired persons, financially disadvantaged groups and those with diminished physical abilities. This directly contributes to the ET 2010 objectives related to accessibility of education and training. Implementation of the LLP also indirectly contributes to the achievement of other objectives of the ET As the report from Estonia argues, for instance, when teachers participate in partnerships or in-service trainings abroad and they learn about new approaches or methods, their own skills and knowledge would improve as a result; and in the end this would have positive impact, for instance, on dealing with problematic students or tackling early school-leaving. But it is clear that the LLP activities do not have direct links to decreasing schoolleaving, increasing the number of graduates in science and technology or attracting more people to lifelong learning. However, the French national report considers that personal international experience better than teachers innovative methods stimulates young leavers motivation and gives a new outlook on their life. Furthermore, people, through direct participation in mobility actions, promote their opportunities of employability. Despite a broad consensus that implementation of the LLP contributes to the achievement of the ET 2010 objectives, several reports (e.g. from Austria, the Flemish Community of Belgium and Malta) argue that it is difficult to measure the scale of the LLP impact. Furthermore, as the report from the Netherlands points out, it is highly ambitious to expect the LLP with its limited budget to contribute substantially to the broader priorities of Education and Training Programme The project leaders from France have some doubts about the longterm effects of their actions, as they believe that when financial support is cut, various initiatives will remain only experimental, without any long-lasting impact. The Netherlands and Slovenia among others point out that the largest impact of the LLP actions is found on the individual and organisational level. However, the impact at systemic level is rather moderate/low. This implies that implementation of the decentralised actions is rather loosely linked with the ET 2010 objectives, which seek changes in the national education and training systems. Summary and recommendations: To sum-up, the Participating Countries argue that implementation of the LLP does contribute to the EU priorities set out in the ET However, systemic measurements of the Programme s impact remain problematic. Furthermore, there are doubts regarding the extent to which the impacts at individual and organisational level translate into change of education and training systems Horizontal policies What specific approaches have been taken within the LLP in your country to meet the Community horizontal policies (the cultural and linguistic diversity within Europe, combating racism, prejudice and xenophobia, making provisions for learners with special needs and for their integration into the mainstream education and training, promoting equality and contributing to combating all forms of discrimination)? To what extent have these approaches been effective? Can any particular points for improvement be identified? 31

32 Within the LLP all programmes should contribute to furthering the horizontal policies of the community. 1 As a result, the NAs dealing with the decentralised actions of the LLP should reflect these principles in their national actions. First of all, by reflecting the European priorities and secondly, if desired, by adding national priorities to the Programme. These priorities should ideally be reflected in a call for proposals, communicated to potential applicants and taken up in the selection procedures, finally leading to selected beneficiaries/projects addressing the horizontal policies. In general, the national reports spend a few sentences on their approaches meeting the Community horizontal policies, often only emphasising that the principle is taken into account without any explanation in what way and providing hardly any evidence on effectiveness of their actions. However, while there are also the reports paying even less attention (e.g. the French and Slovenian national reports), the Flemish Community of Belgium report stands out by devoting much attention to these topics. The national reports mostly include information on specific approaches for approaching difficult-to-reach target groups (by setting the national priorities, devoting extra funding for learners with special needs, steering in selection criteria and monitoring the progress). Other horizontal policies such as promoting cultural and linguistic diversity within Europe, combating racism, prejudice and xenophobia, are often discussed as implicitly taken up in the national actions as a result of participation in mobility actions. Despite rather limited focus, most national reports indicate that the LLP is reasonably effective in meeting the Community horizontal policies. Approaches taken within the LLP to meet the Community horizontal policies. In this particular case, an approach stands for a specific overall strategy or method that is used to give direction to the Programme and potential projects. As stated above, hardly any national reports provide a full overview on the specific approaches to meet the Community horizontal policies, but they often outline general statements or highlight specific elements of their approach. Strategic (setting national priorities and goals). Several national reports explicitly state that the issues addressed by the horizontal policies, promoting equality in particular, have already been laid down in national law/policies before introduction of the LLP, and are therefore implicitly taken into account in the LLP. An example is the Norwegian report, which argues that mainstreaming of horizontal policies is the main feature of Norwegian education and training at all levels, implying a cultural and linguistic diversity, and integration of learners with special needs into ordinary education and training programmes. Another example is the Hungarian report stating that horizontal policies are already recorded in the charter of Tempus Public Foundation. The same counts for the Estonian report referring to the Act on the principle of non-discrimination and equal treatment of all applicants that enforce programmes to monitor coverage of the target groups and inclusion of various subgroups. Finally, an interesting case is the report from the Flemish Community of Belgium indicating that the country is already addressing many horizontal issues in their regular policies, referring to the Kansenbeleid or the policy for equal education opportunities that is one of the cornerstones of the Flemish policy of education and training. Moreover, this report refers to the Steunpunt Inclusief Hoger Onderwijs that was set up in 2008 and is a support Centre for Inclusive higher education to assist students and teachers with disabilities. Finally, they refer to the so-called Encouragement Fund ( Aanmoedigings-fonds ) from which higher education institutions receive extra funding if they take measures to enhance the participation and success rate of youngsters from a disadvantaged background. Some national reports argue that the Community horizontal policies are integrated in the LLP without referring to a specific approach. For instance, the national report from France indicates that the whole Programme stimulates knowledge of cultural diversity and creates consciousness of the necessity of combating discrimination. Another example is the national report from Slovenia, which indicates that there are no specific approaches towards meeting the horizontal policies, but that programmes function in such a way that they achieve these priorities anyway. This is further illustrated by the German national report arguing that the horizontal priorities are seen as 1 General overview of the Lifelong Learning Programme, retrieved at 18 August 2010 from 32

33 integral parts of the projects of the Programme, but are not targeted specifically. Mobility actions and partnerships provide access to foreign cultures and languages, and hence are expected to produce a positive effect on the efforts to combat racism, xenophobia and discrimination, as has been shown by the German research on this topic. In most countries, the participation of learners with special learning needs is stimulated actively in order to increase the inclusion of this target group. Approximately a fourth of the national reports explicitly state that the horizontal policies are taken up as national priority for all or a selection of sectoral programmes. An example is the national report from Bulgaria stating that one of the priorities of Comenius is focusing on children with special educational needs. In Iceland, for example, the NA has used the LLP to raise awareness of the Government to make national priorities in relation to Programme participation more explicit. This has been well received and the disadvantaged groups have become a national priority for The Polish report indicates that the implementation of the horizontal objectives of community policy is one of the priorities of the Erasmus programme. Finally, Romanian authorities also report that the horizontal Community policies are translated into national priorities within the LLP. Concrete instruments (resources, publicity, selection and monitoring). Some national reports indicate that part of the funds was allocated to people with socio-economic or financial difficulties to encourage and include underrepresented groups of people into the LLP. An example in this respect is the Austrian report stating that specific additional funds are available to enable learners with special needs to participate in mobility actions. The report from Bulgaria indicates that the NA annually allocates additional funding within the Erasmus programme for the mobility of students and teachers with special needs. Another example is the case of Greece, where measures were taken to encourage the participation of people from vulnerable groups during the academic year 2009/10. Students with family income less than (attested by the Tax Authority statement) are entitled to the maximum funding rates per month. Some national reports describe that these additional funds are not always coming from the LLP. The national report from Portugal, for example, describes an interesting case that the Government (Ministry of Science, Technology and Higher Education) is destined to facilitate and increase the participation of students with socio-economic difficulties in mobility actions for studies and training courses, offering additional Erasmus scholarships. The Greek report furthermore points out that the institutions participating in the mobility actions receive written instructions according to which each beneficiary with special needs or severe disability is entitled to the maximum amount determined by the European Call for Proposals (while for all other beneficiaries the NA determines a percentage for each year). A few national reports also indicate that during the selection process the NAs allocate additional points to favour underrepresented target groups or types of institutions. For example, the Czech national countries that the NA allocates additional points to favour certain groups of project promoters at the application evaluation stage in line with the national priorities. The Hungarian report also indicates that applicants meeting the criteria set to identify handicaps are not only rewarded by extra points during the evaluation of their tenders, but may also be granted a higher level of support, which serves to compensate for the handicaps. Have these approaches been effective? In order to be effective, the programmes should have contributed to furthering the horizontal policies. There is a divergence in opinions among the national reports on effectiveness of their approaches. However, in general, limited evidence is provided supporting the statements. Learners with special needs The second objective making provisions for learners with special needs and for their integration into the mainstream education and training has received the largest attention during implementation of the projects. The types of policy developments addressing this objective proved to be very diverse, including activities such as: flexible learning pathways in adult and higher education (e.g. the Flemish Community of Belgium); provision of training for people over 45 to avoid social exclusion (e.g. Bulgaria); monitoring of coverage of target groups and 33

34 inclusion of various subgroups (e.g. Estonia); other measures designed to widen access and participation in education from traditionally under-represented groups (e.g. the UK). Although many national reports indicate that the LLP provides a positive contribution for learners with special needs and for their integration into the mainstream education and training, several reports point out that this target group is hard to reach. For example, the reports from the Flemish Community of Belgium and Germany argue that difficulties have been encountered when targeting people in rural areas, migrants and people with disabilities. Reaching people with little or no educational background is also a challenge for the Programme, according to the Austrian report. It indicates that organisation of small-scale events specifically targeted at migrants or people from rural areas proved to be particularly effective in reaching these groups. Regarding gender mainstreaming, the Austrian report mentions that women are strongly overrepresented in the Programme. Several reports (e.g. from Greece, the Netherlands and Slovenia) argue that successful inclusion of people with disabilities is hindered by the lack of funding. Due to increased participation of people from underrepresented groups, the total demand for funding has increased. Hence, the standard amount of funds had to be distributed among a larger group of people who are all entitled for funding. As a result of a shortage in funds, not all demands could be met. Therefore, higher level of funds could contribute to increased effectiveness in this respect. Cultural and linguistic diversity A few national reports indicate that countries have applied specific approaches for the promotion of the cultural and linguistic diversity in Europe. Examples of these projects include a plan for the development of bilingualism and multilingualism in the Brussels Region (the Flemish Community of Belgium); an award of purpose-linked subsidies to all projects that received the European Language Label Award (the Czech Republic); high support of the Erasmus Intensive Language Courses (Greece); and a separate action under the Erasmus sectoral programme - intensive language courses for studying less popular languages with the possibility of additional grants for students with disabilities (Lithuania). However, the majority of the national reports see the mobility actions and partnerships as the most effective instrument for promotion of cultural and lingustic diversity. This is illustrated by the German report stating that the mobility actions have a positive effect on efforts to combat racism, xenophobia and discrimination. Other national reports (including the ones from Denmark, the Flemish Community of Belgium, Ireland, Romania and Spain) also provide very positive assessments regarding the impact of the Programme on promotion of cultural and linguistic diversity. Furthermore, a national survey in Ireland has revealed that the main benefits to learners include the development of softer skills, including improved understanding of inter-cultural issues, enhanced interpersonal skills, greater enthusiasm for learning, heightened ambition and improved ability to work in groups. In Romania the development of competences of communicating in a foreign language, of cross-cultural competences as acceptance, tolerance, fighting discrimination, outrunning the prejudices, cultural aperture, selfaffirmation as representative of the personal culture is specifically outlined. Finally, the Spanish report indicates that the LLP reinforces its European links through mobility and cooperation, promoting among the European Union countries cross-cultural dialogue, language learning and transfer of innovation. However, the majority of the reports also argue that there is a considerable scope for improvement and that the effects on a larger scale are limited. Combating racism prejudice and xenophobia Although much has been done by the NAs to include and support learners with special needs, the Lithuanian report states that the programmes and projects do not pay sufficient attention to combating problems such as lack of tolerance, racism, prejudice and exclusion other than related to disability and discrimination. This 34

35 argument is substantiated by the fact that in all national reports hardly any examples are given of projects aiming at combating these issues. Nevertheless, the Dutch report points out that the international character of the projects still has some indirect influence on people s attitude towards all forms of discrimination. The participation in mobility actions is seen as contributing significantly to combating racism and prejudice, as well as promoting cultural and linguistic diversity. Summary and recommendations To conclude, the implementation of the LLP certainly contributed to the achievement of horizontal priorities. However, further progress in this area is necessary. The national reports provide the following suggestions: More targeted approach towards a subset of horizontal priorities is desirable, as the priorities are often not explicitly mentioned within the activities (see reports from the Netherlands and Poland). Furthermore, more attention should be paid to the specific objectives related to the horizontal policies that are relevant to the national context during the application and selection procedures. The amount and division of funds should be reconsidered. The increased participation of the underrepresented groups has led to an increased demand for funds. Promotion of tolerance and prevention of prejudice should be prioritised for at least several years, since there is a lack of projects specifically directed at combating racism, prejudice and exclusion. This recommendation was expressed in a report from Lithuania Dissemination and exploitation of results What system and tools do you use for disseminating and exploiting the LLP results in your country? Are they fully exploited? Where can you see the possibilities for improvements? All activities and projects funded by the EU should be given as much publicity as possible to potential users. 2 Effective dissemination, as in providing information in a planned way to a relevant audience, and exploitation of the sectoral programmes should reach all relevant target groups and increase the number of participants in the Programme. Generally, the national reports state that a variety of tools are available and used for dissemination, but exploitation of the results remains problematic. Usually, exploitation and exchange of the results on the different levels does not take place. More coordination at the European level appears to be desirable. Most national reports indicate that much more could still be done concerning dissemination and exploitation of the results. Tools and systems used for dissemination and exploitation. Various tools have been used for dissemination and exploitation of the results of the LLP. The most frequently mentioned tools include the following: 1. Conferences, meetings, seminars and workshops (e.g. AT, BG, CY, DE, EE, EL, ES, FI, FR, HU, IS, LT, LU, MT, NL, NO, PL, PT, SI, TR, UK). 2. Internet websites (e.g. AT, BE (nl), BE (fr), BG, CY, DE, DK, EE, FI, FR, HU, IS, IT, LT, MT, NO, PL, SI, TR, UK). 3. Publication in magazines, newspapers and newsletters (e.g. AT, BG, DE, EE, EL, FI, FR, HU, IT, LT, MT, NO, PL; translations in RO, TR, UK). 4. Written dissemination material (e.g. AT, CY, DK, EE, FR, IS, MT, NL, NO, RO, SI, UK). 5. Participation in local, regional and national media (e.g. AT, BG, CZ, EL, FI, HU, IS, LT). 2 Dissemination and exploitation of the results, retrieved at 19 August 2010 from ttp://ec.europa.eu/education/lifelong-learningprogramme/doc98_en.htm. 35

36 6. Annual reports (e.g. BG, IS, LU, NL, NO, SI). 7. Databases (e.g. EE, EL, IT, LU, PL, TR). 8. Interim impact- assessment procedures and follow-up visits (e.g. BE (fr), BE (nl), DE, EL, ES, IT, LU). 9. Social networks (e.g. BG, EE, EL, IT, LT). 10. Expositions and exhibitions (e.g. BE (fr), BE (nl), BG, HU, LT). 11. Word of mouth (e.g. BG, EE, FI, NL). 12. Information centres (e.g. AT, EL, IS). As shown above, the Internet and various events are the most frequently adopted tools for dissemination. Events, such as conferences, meetings, seminars and workshops, are used for the promotion of the programmes, exchange of the best practices and for information gathering. As thematic events can be organised for specific target groups, it is easier to convey the right message and to the relevant target group. The Internet mostly concerns the website of the NA where general information on the Programme, its projects and results are given, but it also concerns websites of partner institutes, educational portals, blogs and online publications. However, only some reports indicate that there is a systemic approach towards dissemination and exploitation. The beneficiaries share approach has been introduced in Bulgaria. This approach focuses on the exchange of experiences and the best practices during different kinds of events. The national reports from Belgium, the Netherlands and the UK also stress the importance of sharing best practices. The French report points out to have implemented a transversal approach by means of thematic events and representatives of different projects. Some Participating Countries, such as the Czech Republic, France, Hungary, Norway and Spain, point out to have assigned a coordinating institution responsible for dissemination of the results, such as a Knowledge Management Group in Hungary. Other Participating Countries seem to disseminate the project results in a less coordinated way. Exploitation level. Concerning the level of exploitation, several impact levels can be distinguished, such as the reputation of the LLP and sectoral programmes, dissemination of information and results of the programmes towards the target groups, but also communication between the different programmes and project managers. Most of the dissemination activities that have been mentioned in the reports are directed at informing potential beneficiaries and individual participants in particular. There is a divergence in opinions regarding the level of effectiveness in terms of reaching the target groups with information on the LLP. Several reports (e.g. from the Czech Republic, the Flemish Community of Belgium and Iceland) argue that the majority of target groups are reached. However, this does not always lead to an increase in application/participation numbers. Other reports, such as the Czech and Irish report, indicate that dissemination could have been better, as not all target groups are familiar with the Programme. For instance, the Comenius programme is not very well known among Irish teachers. The Czech report indicates that the level of dissemination and exploitation also differs per region. Not all regions are sufficiently involved in the LLP as a result of low dissemination activities. Therefore, these regions are specifically targeted by the Czech NA. Consequently, the reputation of mainly the sectoral programmes also remains to be improved. Although reports from, for instance, the Flemish Community of Belgium, Iceland and the UK are quite positive about the implemented tools for dissemination and the target groups reached, they contain limited evidence of the achieved results. The lack of concrete evidence of achieved results makes objective evaluation of the level of exploitation difficult. According to all national reports, dissemination of both information on the Programme and the results achieved has been taking place, but the reports from Lithuania, the Netherlands, Norway and Poland point out that real exploitation is rather limited. Generally, the statements on the level of exploitation are hardly made in any of the reports. The national reports indicate that successful exploitation is hindered by the following obstacles: 36

37 According to the reports from Portugal and Slovenia, before the start of the Programme not enough time and resources were available for the creation of a systematic and structural approach for dissemination and exploitation of the results. But the reports do not specify in detail how this hampers dissemination and exploitation. A wide range of target groups makes dissemination and exploitation on the Programme level challenging. According to the Norwegian report, fully satisfactory results regarding dissemination and exploitation are hard to obtain due to diversity of target groups. Furthermore, the Czech and Flemish Community of Belgium reports indicate that participation of certain groups remains to be improved. For instance, the Flemish Community of Belgium report particularly mentions the limited involvement of special schools, Pupil Guidance Centres, parents associations in case of Comenius, and students with disabilities as well as adult combining work and study in case of Erasmus. The Flemish Community of Belgium report indicates that during the implementation of the Programme, monitoring is used as a tool to disseminate results among end-users of different programmes. However, although the report does not elaborate on the specific cause, it also points out that information available is hardly really exploited. There exists a large gap between the projects that are being carried out and the policymakers at the sectoral or national level. The Dutch report stresses that this makes it difficult to mainstream the results in national, sectoral and institutional policy developments. On the project level, there are often enormous differences in efforts made by project managers to disseminate and exploit results. The Danish report indicates that some project managers make quite a lot of efforts to stimulate beneficiaries to share experiences, while others are less goal-oriented. According to the Flemish Community of Belgium report, dissemination towards other organisations hardly takes place. Due to duration of the programmes and projects the position of project coordinator becomes vacant and is sometimes taken up by different persons. However, according to the report from the Flemish Community of Belgium, potential new coordinators often lack skills and expertise in international projects because crucial information is hardly communicated between the different projects. After the finalisation of a project, there is a lack of feedback from the NA on the final report, as the Czech report argues. There is no clearly defined person or organisation responsible for dissemination and exploitation after a project has been finalised. As a consequence of these difficulties, the Participating Countries are not able to fully exploit the results available. However, the national reports recognise the relevance of dissemination and exploitation in order to increase the effectiveness of the Programme. Most of them agree that more could be done to increase the visibility of the LLP and to promote participation. Summary and recommendations: To summarise, there are considerable efforts aimed at dissemination and exploitation. Nevertheless, the Participating Countries argue that this area remains one of the weaker links of implementation of the LLP. The reports contain a wide range of recommendations. The majority of them are directly related to introduction of a more systematic approach towards dissemination and exploitation as well as increased top-down coordination. Specific points for improvement of effectiveness of dissemination and exploitation activities on the different levels of the LLP (centralised actions, decentralised actions, sectoral programme and project level) include the following: Dissemination and exploitation of results should be more systematically organised at the European level (report from Lithuania). More attention could be paid to the exploitation of the results in comparison to the current level of attention given to cooperation and participation in the process. Consider development of a single European database with up-to-date statistics and best practices that would be widely accessible. This recommendation is endorsed in the reports from Lithuania Norway and Portugal. 37

38 Top-down and bottom-up dissemination strategies and practices could be better combined (see reports from Greece and the Netherlands). In other words, coordination between centralised and decentralised actions could be fine-tuned, and closer cooperation between all levels could be stimulated as well. Consider more active involvement of key policy makers and public officials in the dissemination activities. This recommendation was endorsed in the report from the Flemish Community of Belgium. There is a need for better guidance of the project managers, according to the Dutch report. The guidance should focus on provision of advice regarding what should be achieved and how this should be done. It is preferable that such guidance would be communicated before the start of the project. Consider integration of different websites and web-based databases. The Czech report argues that this should improve the exploitation of project outcomes Visibility of the LLP To what extent is the LLP known to the education and training community in your country (both the LLP as such as well as its component parts)? Overall, the national reports consider that awareness of the LLP brand is lower than awareness of the brands of each sectoral programme. The level of knowledge about the LLP is low among education experts, public officials and SMEs (abbr. of small and medium enterprises). However, the knowledge about separate sectoral programmes is considerably higher than the whole LLP. Although these programmes are quite well recognised, they are not always associated with the LLP. This is due to the fact that specific programmes have been running for a longer time than the integrated Programme and have already established themselves as sustainable brands. Furthermore, the LLP beneficiaries are the most familiar with the programmes and actions, where they directly participate, and considerably less knowledgeable about the other sectoral programmes (e.g. BE (nl), BE (fr), CZ, FR, HU, NL, SE and SI among others). The national reports employ the following criteria for assessing awareness about the LLP and its sectoral programmes: The number of applications (e.g. the Czech Republic, Estonia, Lithuania and Romania among others). Participation of new target groups (e.g. Bulgaria, the Czech Republic and Lithuania). The number of visits to the website of the National Agency (e.g. Lithuania). Number of media reports concerning lifelong learning (e.g. Austria). The reports argue that the main obstacles to higher awareness of the LLP include the following issues: The title Lifelong Learning Programme is considered to be misleading, since it is wrongly associated with the lifelong learning of adults (e.g. in the national reports from the Flemish Community of Belgium, the Netherlands and Slovenia). Moreover, Turkey notices that the Programme s name is similar to a television brand and it is hard to pronounce properly. The project promoters and/or participants are not aware of the link between the LLP and its component programmes. They are still often believed to be separate interventions (e.g. the Czech Republic, France, the Netherlands). The complexity of the Programme. The national report from Hungary summarises: < > People who have been involved with programme implementation for years sometimes confess that it took them ages to understand this multifarious and complex structure and would not dare to declare that they know the programme inside and out. And this is typical despite the many simplifications introduced during the integration. 38

39 The title Socrates is still used instead of the LLP (e.g. in the Czech Republic, Finland and the Frenchspeaking Community of Belgium among others). In some cases, limited resources available for a sectoral programme make it difficult for them to stand as a strong brand (e.g. the Flemish Community of Belgium, Slovenia). The following paragraphs concentrate on awareness of each sectoral programme. Comenius is a sufficiently well-known programme, as the numbers of filled applications usually exceed the numbers of available places. The kindergartens are the worst-informed group: the number of applications from this target group is relatively small (e.g. in Norway). In the Czech Republic, applications from kindergartens comprise only 2%, whereas in Bulgaria, Comenius has become particularly popular among kindergartens. Moreover, certain target groups are not aware that they are eligible for funding under the Comenius programme. For instance, in France, language teachers usually do not know about such opportunities. Despite positive assessments of awareness of the Comenius programme in the majority of the Participating Countries, the reports from the Flemish Community of Belgium, Ireland and Turkey claim that the potential participants feel a considerable lack of information about this programme. Erasmus is nearly unanimously considered to be the most visible and the best-known international and mobility programme of all. The national reports argue that information about Erasmus reaches its target groups without any hindrances. However, further progress is needed in increasing the awareness of several specific target groups. For instance, the national report from Cyprus states: The Erasmus project in 2009 seems not to have had much success in attracting staff from enterprises and organisations. In addition, Finland argues that the knowledge of the staff mobility for training could be improved. Generally, the awareness about the Leonardo da Vinci programme among the participants is quite good. However, several specific target groups have a lower level of knowledge about this sectoral programme. According to the national report from the Flemish Community of Belgium, very few employers, workers and VET students are familiar with the programme, its possibilities and the benefits it may bring. Moreover, low awareness level correlates with unsatisfactory participation level in the programme. In addition, Poland and Portugal point out that SMEs are particularly poorly informed about the opportunities offered by the LLP. This hinders potential cooperation between vocational education and enterprises. Grundtvig is not very well-known programme among citizens. It is more popular among the relevant professionals. However, as a result of the new Grundtvig Workshops the information seems to reach the general public as well (e.g. the Czech Republic among others). In addition, sometimes an idea of applying e.g. for structural funds seems more attractive for the beneficiaries, since there are no requirements of international cooperation, thus, language barriers are eliminated (see the national report from Poland). Low awareness of Grundtvig is linked with several factors: a) its wide and fragmented target groups; b) small size; c) its lack of linkages between Grundtvig activities and the formal education system; d) Grundtvig is a relatively recent programme (e.g. the national reports from France, Luxembourg and Norway among others). On the other hand, Grundtvig has gained extremely high popularity in Cyprus and Finland. For instance, funds from other sectoral programme s budgets were transferred towards Grundtvig in Cyprus. Summary and recommendations: To sum-up, the sectoral programmes (with the small exception of Grundtvig) have well established brands and their name recognition among potential target groups and the professionals working in the field is high. The visibility and awareness of the LLP is also gaining momentum. The national reports also provide the following recommendations for further improvement: There is a need for a more firmly rooted communication strategy of the LLP (e.g. the national reports from the Czech Republic, France and Norway). Such demand stems from the fact that the structure of 39

40 activities is excessively complex. As the Czech Republic argues, it is difficult to comprehend the structure for those who have not yet had an experience with the programme. Hungary also admits with the extreme complexity of the programme. Media should be more involved in highlighting the successful projects (report from Bulgaria). Major effort should be made to find a new attractive brand name for the Programme (the Flemish Community of Belgium). Reducing segmentation of the sectoral programmes and increased emphasis on the lifelong learning should facilitate awareness-raising initiatives (the Flemish Community of Belgium). There is a need for a special information strategy, organisation of special seminars for managerial staff in schooling, project managers, and informal meetings with the successful participants of already realised partnerships and mobilities (e.g. in Finland, Portugal, Slovakia and to some extent in France). Increased funding of the sub-programme and support to the networks in the field of adult education should contribute to higher visibility of Grundtvig (e.g. Finland). 3. Efficiency of the lifelong learning programme 3.1. Cooperation between the main implementation bodies, management workload and efficiency of monitoring mechanisms To what extent is the system of cooperation between the Commission, Executive Agency, National Agencies, National Authorities, and LLP Committee efficient and well functioning from the point of view of your country? What is the degree of complexity and level of the management workload? Are the monitoring/supervision mechanisms applied by the Commission towards National Agencies efficient/cost effective? What are the areas for improvement considering the need for a smooth implementation of the LLP? The system of cooperation between the DG EAC, the EACEA, the NAs, the National Authorities and the LLP Committee. Generally, the system of cooperation among the NAs, the National Authorities and the LLP Committee is considered to be well functioning and fairly efficient. However, the Participating Countries point out that relationship with the EACEA and to some extent with the DG EAC is complicated. The problem of insufficient accessibility to the EACEA from the point of view of the NAs is emphasised in the vast majority of the national reports. Dissatisfaction is expressed in various statements, e.g. the weakest link in the system of cooperation (Estonia); the EACEA is not providing enough information about centralised projects, their outcomes and the Flemish participants in such projects (the Flemish Community of Belgium); the relationship < > has been very limited and formal <...>, there seems to be a lack of the collegial relationships <...>, there is virtually no cooperation between the NA and the EACEA, and the NA does not automatically receive information on Icelandic applicants or projects (Ireland); Cooperation <...> is still in its early stages and is not always smooth (Lithuania); the relationship with the EACEA is < > non-existent. There is no structural exchange of experiences and knowledge (the Netherlands); Cooperation <...> is seen as too bureaucratic and autocratic, not considering enough the opinions and suggestions of the National Agencies and committees (Slovenia). In addition, several countries (e.g. ES, IT, MT, PL, RO and UK) also point out that currently there is no synergy between the EACEA and the NA (in Polish case, the Grundtvig programme is excluded from negative evaluations). Just in the minority of the national reports the cooperation with the EACEA is described as more supporting than hindering (e.g. cases of the Czech Republic, Greece, Norway). Such a communicative gap could emerge due to divided management for centralised actions (e.g. Estonia) or due to the fact that specific needs vary according to the National Agencies, thus, not all attitudes are compatible and not all decisions are satisfactory (e.g. Slovenia). 40

41 The attitudes towards the relationship with the DG EAC are twofold. On the one hand, the relationship is seen as flexible and helpful, since the Commission reacts urgently even to non-standard situations and gives practical advice (e.g. CZ, DK, EL, ESFI, IS, LT, MT, NO, PL, RO). On the other hand, the Flemish Community of Belgium argues that cooperation with the DG EAC (and the EACEA) is hindered by heavy administrative, control and reporting requirements, part of which are considered to be useless and an unnecessary administrative burden. Ireland adds that the LLP implementation is obscured by the time lag in receiving the NA s contract from the DG EAC. The Netherlands state that the Commission does not always deliver the guidelines and application forms in a timely manner, which causes feelings of uncertainty on the part of the National Agency and the educational sector. Sweden would also like to see more timely feedback from the Commission to the NA. The cooperation within the LLP Committee is smooth and works well (see the national reports from the CZ, ES, IS, LT, NO, PL, RO). Usually, it is composed of officials delegated by public authorities. Thus, its decisions are in a line with the policies implemented by the national Government. The cooperation between the NAs and the National Authorities is assessed as functional (e.g. the BE (nl), CZ, EE, EL, ES, FI, IS, MT, NO, PL, PT, RO), sometimes satisfactory (the Netherlands). However, it took some time to adjust to the redefined roles and tasks, but progress in this area is considerable. The relationship between the NAs is viewed as very good (e.g. see the national reports from CY, ES, FI, HU, NO, PL, UK) or satisfactory (e.g. in the case of the Flemish Community of Belgium). The Flemish Community of Belgium report argues that there is room for improvement, e.g. under the form of short or long shadowing periods of administrators in view of the professional development of the NA administrators, or by developing projects together ). In conclusion, the cooperation among institutions at the national level is considered to be more efficient and intense than at the European level. Degree of complexity and management workload level. Overall, the national reports argue that the degree of complexity and the level of the management workload is high (e.g. see the national reports from the BE (nl), CZ, EE, ES, FI, HU, IE, LT, NL, PL, PT, RO). A number of the Participating Countries claim that the administrative burden has increased after introduction of the LLP, although it originally aimed at simplification and transparency of the management procedures (e.g. Estonia, Finland, Hungary and Portugal among others). However, the national reports from Denmark and the French-speaking Community of Belgium point out that initially introduction of new procedures and forms has caused higher workload, which considerably reduced during subsequent years. The reports argue that the highest workload is related with the following processes: The process of reporting to the DG EAC (e.g. BE (nl), CZ, DK, EL, ES, FI, IE, LT, PL, RO, SE, TR, UK). The largest bureaucratic obstacle relates to high number of various documents, e.g. manuals, memos, financial reports, containing various parameters, which have to be produced by the NAs. The factors, which cause the reporting workload, are as follows: Duplication of work. For instance, the preparation of the Annual Report is time-consuming, as the same information can be found in LLP-Link and other websites (see the national reports from Spain and Sweden among others). Unsuitable timing to present the reports. For example, the period of the highest intensity Call cycle coincides with the reporting period to the DG EAC. Thus, it would be reasonable to separate the deadline for calls for applications from the deadline of the Annual Reports (e.g. Poland). Spain argues that there is a greater workload between mid-october and the end of June when additional tasks are added to the normal activities, including training of new beneficiaries, preparation and publication of the calls for proposals, preparation of annual reports, and 41

42 evaluation of proposals and contracts. The national report from Ireland also adds: Differences in the academic years between institutions have tended to complicate the reporting process and in some instances Irish HEIs have had to second guess results not available in partner institutions. Lastly, Austria and France argue that the deadlines set for the NAs are mostly too short. Disproportional reporting requirements. For instance, financial reporting requires including a lot of parameters; moreover, narrative descriptions have become more extensive. Considering insufficient time and financial resources, these requirements are too excessive (e.g. the national reports from Denmark, Sweden and Turkey). Although procedures are demanding, the feedback from the Commission is quite limited and insufficiently specific. France gives an illustration: it took nine months for the Commission to present the assessment of French Declaration of Assurance and five months to present a preliminary auditing report. A cumbersome structure of the Programme from the applicants point of view (e.g. see the national reports from the BE (fr), CZ, EE, ES, FI, IE, LT, RO and TR). For instance, each action of every sectoral programme used to be funded and managed according to different rules; introduction of electronic application forms is too slow and sometimes complicated; grant application procedures are also seen as overly complex. The amount of sudden changes regarding e.g. rules, guidelines for NAs, application procedures and management tools, introduced by the DG EAC (e.g. CZ, EE, FI, HU, LU, PT and UK among others). Austria and France point out that due to such frequent changes of the guidelines, contracts and reports, the amount of translation-related work considerably increases. Thus, it consumes additional financial and labour resources. The delayed publications of certain documents, which hinder the implementation process of the whole Programme (e.g. CZ, DE, HU, LT, NL, PT, SE and UK among others). For instance, France and the UK argue that the new Guidelines for NAs usually come too late (at the end of December), and it becomes impossible for the NAs to plan their activities effectively. Financial burden. As a consequence, a number of staff working in the NAs is insufficient, thus, it is difficult to fulfil all of the requirements (e.g. see the national reports from Slovenia and Turkey). Diversity of the target groups and large number of actions in the sectoral programmes (e.g. Finland, Germany and Poland among others). Denmark, Lithuania, Sweden and Turkey among others argue that the formal procedures and various technical tasks are so extensive that they unduly use the resources, which could be allocated to the contents of the actions funded by the LLP (including). Are the monitoring/supervision mechanisms applied by the Commission towards National Agencies efficient/cost effective? Overall, efficiency of the monitoring and supervision mechanisms has increased throughout the implementation of the LLP. However, the opinions, regarding the extent to which the monitoring and supervision requirements are efficient, diverged. On the one hand, Cyprus, Greece, Norway, Romania and Turkey claim that the processes are rather efficient. The most efficient monitoring and supervision instruments include the reports (e.g. Bulgaria, Lithuania and Turkey), external auditing (e.g. Sweden, Turkey) and monitoring visits (e.g. Turkey). On the other hand, BE (nl), DE, DK, FI, IS, LU, PL, NO, RO, SI and the UK argue that reporting requirements are too excessive and unduly take too much resources. Furthermore, auditing is too frequent: it is not necessary to audit NA s work procedures on an annual basis simply because they do not change very much from one year to the next (see the national report from Iceland). Furthermore, Austria points out: there continues to be too much control of NAs, which leads to unnecessarily intensive administrative procedures < >. The monitoring and control system < > requires too many resources from the NA and is not proportional and justified in the sense of Article 6 of the Decision establishing the LLP. Luxembourg assesses that the complexity of management entails an enormous workload that seems disproportionate to the size of agencies. Germany also argues that the demands of the Commission s monitoring and control system are seen as inefficient and producing too much administrative demands. According to the national report from France, efficiency decreases due to a large 42

43 number of control mechanisms, moreover, they are arbitrary and ineffective. What is more, Spain and Sweden claim that the preparation of the Annual Report leads to unnecessary duplication of work, since the same information can be found in LLP-Link and other websites. Summary and recommendations To sum-up, the relationships between the NAs, Authorities and the LLP Committee are efficient and well functioning. The relationships with the Commission, in the view of several national reports could be improved, while the relationships between the NAs and EACEA deserve specific attention. Furthermore, the reports argue that the degree of complexity and the level of the management workload is high and management/supervision mechanisms are too excessive and require too much resources. Nevertheless, the cost-efficiency of the latter mechanisms has increased between 2007 and The Participating Countries provided a number of recommendations for a smooth implementation of the LLP. They can be divided into two basic areas: a) management and b) monitoring. Recommendations regarding improvements in management of the LLP: Reduce administrative burden and stabilise the procedures (e.g. minimise the amount of changes introduced by the DG EAC) and simplify them as much as possible, e.g. introduce uniform funding procedures for similar activities (e.g. BE (nl), CZ, DK, EL, ES, FI, FR, HU, LT, LU, NL, RO, SI, TR and UK). Improve electronic management tools. This should include: faster completion of electronic application forms, improvement of databases, introduction of electronic signature etc. (e.g. EE, ES, FI, IE, LT, NL, SI, TR). Establish an information channel between the EACEA and the NAs in order to strengthen cooperation and improve the quality of the LLP implementation (e.g. CZ, DE, ES, FI, IS, IT, LT, MT, NL). Simplify the Programme management from the point of view of beneficiaries (e.g. BE (fr), CZ, ES, FI, NL, NO, PL, RO, TR). This could include simplification of application forms, introduction of framework contracts for the institutional beneficiaries, who have produced good results in the past, lower administrative burden of implementation of projects (e.g. further consider introduction of lump-sum payments). Organise informal meetings of the NAs from all the Participating Countries or the directors of the implemented projects. This should facilitate peer-learning, transfer of innovations and good practices (e.g. BG, EE, ES, FI, IS, PT, RO, UK). Timely provision of implementation documentation to the NAs (e.g. reporting requirements for the NAs, application forms, etc.). It is vital for the smooth coordination and implementation of the LLP (see the national reports from DK, EE, FI, NL and UK). Further progress with introducing lump-sums payments is necessary (e.g. AT, FI, IE, LT, RO, UK). Allocate more financial resources for the management of the LLP (e.g. Finland, Turkey). Easier procedures for reallocating funds within the sectoral programmes should be applied. More decision making autonomy in this area should be delegated to the Participating Countries (e.g. EE, EL, FI, LU, NO, SE, UK). The complexity of management should be proportionate to the size of the National Agency. The national reports from France, Luxembourg and Malta among others argue that smaller agencies must cope with an enormous workload, which is disproportionate to their size. Strategic decisions on each sectoral programme should be discussed well in advance (e.g. the UK). Recommendations regarding monitoring of the LLP: The mechanism of monitoring should be reviewed and reduced (its scope should be minimised especially during the periods of e.g. Declaration of Assurance finalisation) (e.g. BE (nl), CZ, EE, ES, HU, IS, LT, PL, SE). Slovakia adds that monitoring should be more focused on the outputs rather than on the minor details of procedures. 43

44 National auditing measures should not duplicate European-level measures (e.g. Spain and Sweden among others). Inefficiency could be reduced by giving for the NAs a higher status of ISO-certification (e.g. Germany and Luxembourg) Divided management responsibilities for centralised and decentralised actions Does the system of divided management responsibilities for centralised and decentralised actions simplify the LLP implementation or does it make it more complicated in comparison to the previous programmes' set-up? Overall, the national reports do not provide a straightforward answer to this question: the vast majority of the countries point out both positive and negative aspects of the divided management system. The Participating Countries consider that system is more beneficial for the projects that are implemented under the decentralised actions, but also leads to coordination problems. On the one hand, such division of responsibilities allows the NAs to focus directly on small and medium-size projects and networks, less extensive activities and their implementation at regional and local levels (e.g. see the national reports from the Czech Republic, the Flemish Community of Belgium, Malta and Spain among others). In addition, involvement of the EACEA in the management of the centralised actions helps to meet European-wide priorities and decrease the workload of the NAs (e.g. in Estonia, Lithuania, Malta, Poland, Slovenia). The UK points out that educational and training policies differ from country-to-country, thus, decentralised model provides each participating country with a possibility to act according to the national context. On the other hand, the system of divided management responsibilities has several disadvantages: It creates artificial barrier between the centralised and decentralised actions and leads to miscommunication: the NAs feel a lack of information about centralised actions. For instance, France argues that the NA does not receive information from EACEA about the national Higher Education Institutions (HEIs) participating in centralised actions. Furthermore, the NAs, due to the lack of information, cannot properly inform and guide potential beneficiaries, who are automatically directed to the EACEA. There is a lack of structured cooperation between the NAs and EACEA, thus, the beneficiaries usually feel unsatisfied (e.g. BE (nl), BE (fr), DE, DK, EE, ES, FI, FR, HU, IS, LT, LU, PL, SI, SK, TR). The participation level in the centralised actions is diminishing partly due to the fact that the NAs are not allowed to provide potential applicants with information (e.g. including Estonia, Norway and Spain). A concern due to sustainability and relevance of the centralised projects is expressed, since the NAs became less involved in the evaluation process of the centralised actions. Thus, it is unclear who would assess the importance of the projects in the national contexts. There is also a risk that decentralised and centralised projects do not complement each other or there are overlaps due to insufficient exchange of information (e.g. including Estonia, Iceland, Luxembourg, Malta and the Netherlands). Information about centralised actions on the DG EAC website is presented in a complicated way. Moreover, some important information e.g. about partner organisations of the centralised projects is not published at all (e.g. see the national reports from Estonia, the Flemish Community of Belgium and Sweden). Summary and recommendations: The Participating Countries agree that there should be more communication between centralised and decentralised actions. Such common consensus stems from the current situation: there hardly exists coherent communication between the NAs and the EACEA (e.g. see the national reports from BE (fr), CZ, DE, ES, FI, FR, IS, 44

45 IT, LT, MT, NL). Finland also notes that currently there are too much differing practices and working cultures in both the decentralised and centralised actions and little possibilities for different target groups to cooperate with each others. It is strongly advised to reduce such inefficiencies within the Programme Management supporting tools To what extent are the management supporting tools (e.g. LLPLink, EST, Nety, Circa) adequate for the efficient management and implementation of the programme in your country? Give examples of the specific management tools used in your country and of their contribution to the effective LLP management. Overall, the Participating Countries provide a critical assessment of the management-support tools and argue that they have not yet produced the expected benefits. An important factor hindering efficiency is a fact that some management-support tools are still at the preparatory stages, for instance, EST (European Shared Treasure), ADAM (Advanced Data Archive and Management System) and EVE (Espace d'échange virtuel). This inevitably implies that the NAs employ a dual (parallel) system for collecting and managing the information. The Participating Countries also feel that the technological requirements for the operation of the IT systems are increasing. Thus, the staff should be provided with more training in the area of IT innovations (e.g. see the national reports from DK, EE, IE, LT, LU and PT among others). Comments concerning particular management support tools Overall, the Participating Countries feel that the LLPLink does not deliver the expected benefits. As it is still incomplete, it creates administrative burden (developing financial statistical reports and managing projects as well as applications), but it is not useful for the purposes of monitoring and decision-taking (see the national reports from CZ, DE, EE, FI, HU and PL). For example, Austria argues that the LLPLink can be used as data storage, but not as a statistical tool. In addition, technical difficulties cause serious problems for the applicants (while filling in applications online) and coordinators of the NAs (while cross-checking documents) (e.g. Lithuania). The report from Iceland is particularly critical: The managerial and technical incompetence shown by the responsible unit at the Commission is beyond belief. No NA would ever be permitted by the EU to show such dismal results and ineffective management. The report from the Netherlands also argues that the use of the administrative tool LLPlink has not been developed towards its full potential. Most administrators are having difficulties in using it and the system is at this point not very robust and stable. All organisations responsible for their sub programmes have therefore duplicate book keepings, which are not integrated in one bookkeeping system for the whole LLP. <...> the National Agency uses a shadow bookkeeping system to monitor the administrative and financial progress of the sub programmes. The national reports from the Netherlands and Slovenia argue that NAs cannot draw data from the system, thus, it is unable to use that data for internal management purposes. Furthermore, the system is unreliable, not user-friendly and very often modified (e.g. including BE (nl), DK, EL, ES, LU, NL, PL, PT, RO and UK). Sometimes the data have to be exported manually into the reports and this causes additional workload. To some extent this problem was overcome by introducing online applications and reporting forms. The communication tool Circa aimed to replace Nety. Several Participating Countries are quite satisfied with this tool (including AT, BE (fr), BG, DE, DK, SI). However, other countries argue that Nety functioned much better and was more user-friendly. For instance, Circa does not provide any possibilities to cooperate and communicate with foreign colleagues from the NAs. As a result, the value to the users of this tool has been lost (e.g. the Czech Republic, Estonia, Greece, Lithuania and Poland) and the NAs have to use a shadow bookkeeping system. 45

46 However, Circa is useful for finding and saving all the required documents (in Bulgaria, the Czech Republic, Lithuania, Spain). But usually it is unclear where these documents can be found, as there is a lack of clear title, to which a particular notification refers (e.g. see the national reports from Germany, Greece, the Netherlands and Poland). Moreover, France points out that Circa contains a lot of information that does not concern the administrators of the Programme and, on the contrary, lacks very important one. Nevertheless, Denmark argues that it is really helpful, if a user properly knows how to use it: Once we got the hang of the system, it is really smart. EST database has a huge potential, but it cannot function fully it is still at a preparatory stage, coping with technical difficulties (e.g. CZ, DE, EL, ES, FR, LT, NL and UK). Romania adds that EST remains mostly an internal instrument. According to the national report from Austria, EST is not a useful tool at all, as the data entry is complicated and representation of the data is unreliable. ADAM is at a preparatory stage, thus, it is hard to evaluate it properly (e.g. the Czech Republic). The advantage of this database is that data about the projects are registered immediately after contracting (e.g. Poland). Spain and the UK evaluate ADAM as advantageous as well: it provides benefits for contracting and partner organisations in viewing information and offering additional dissemination opportunities. According to Austria, ADAM has a high potential with little administrative requirements. However, Romania argues that ADAM is known mainly internally, thus, it can hardly contribute to the dissemination of results. EVE is at a preparatory stage, thus, it is hard to evaluate it properly (e.g. see the national reports from the Czech Republic, France and the UK among others). However, the Czech Republic points out the strength of this tool: < > EVE, due to its transversal nature, entails good prospects in terms of successful dissemination of the results of the projects of the LLP. Nevertheless, it still lacks user-friendliness (e.g. see the national report from Austria). Some other specific management tools are worth mentioning, since they contribute to the effective LLP management in the national contexts: A nationally constructed electronic reporting system, which is used by the Erasmus project coordinators in Finland. DBH system (Database on Higher Education), which contains statistical information on the results of all student exchange programmes and schemes in higher education (Norway). ECAS virtual forum for communication between the NAs and the DG EAC (Estonia). However, it has a very low level of user-friendliness. Erasmus-Net, which facilitates monitoring and communication (established by NA in Turkey). etwinning, as it is a useful tool for finding new project partners. However, its usability is quite poor (e.g. Finland). Euroguidance information network used when working with specific target groups such as counsellors and decision-makers; the tool is also used to promote European cooperation, to disseminate knowledge and to promote generally the quality of European education (e.g. Sweden). Eurydice (e.g. Romania, Sweden). FiCop, the Exceptions Register, the Problems Register, and the Beneficiaries Register with difficulties in implementing the projects (tools used in Romania). FINERA a tool of managing the project inside the Foundation (for Erasmus in Poland). GISE Erasmus Students Information Exchange (Poland). KVASS the quality monitoring programme used in Norway. Leopass (Romania, Spain). Microsoft Excel, which is employed to complement LLPLink (in Finland, the French-speaking Community of Belgium, Greece, Malta, Norway). M-tool Project (Portugal). Multipass (the Czech Republic). 46

47 NA Talk paneuropean platform of information exchange between NAs (for Erasmus in Poland). NA website, which is used for communication, monitoring and information sharing among stakeholders (in Turkey). Pandora (the Czech Republic). Rap4Leo reporting database, which provides information about beneficiaries, implementation of the projects and the feedback about mobility from various perspectives, e.g. budget, schedule etc. (in Finland, Poland, Portugal, Turkey). Ulysee and Pénélope (in France). A part from the aforementioned software is constructed by national institutions. It helps to cope with various technical difficulties, which arose due to the fact that some pan-european management supporting tools are still at a preparatory stage. The reports also provide the following criticisms regarding the management supporting tools: Duplication of work. For instance, Estonia argues that in addition to the LLPLink in Erasmus, there exists a separate management tool NA reporting software. These two different systems require additional resources and hinder efficient programme implementation. Finland and Germany state that in order to complement the LLPLink, staff must use Excel and other IT tools. This causes extra and overlapping work. Denmark argues that project managers still have to mail application forms in paper and to fill out the electronic form. Creating separate systems for disseminating project results. For example, ADAM database is supplemented with EST and EVE. Although these databases disseminate results of different projects, it is unclear why one well-functioning database cannot be exploited for wider purposes and cannot cover all the sectoral programmes (see the national reports from Austria, Estonia and Germany). The lack of stability in the tools. Sometimes it is questionable why well-functioning and user-friendly instrument is replaced with inefficient one, as in the case with Nety and Circa (e.g. the Flemish Community of Belgium, Slovenia). Summary and recommendations: In summary, management-support tools should be further development and consolidated. Specific recommendations for further improvements include: The NAs staff should be provided with more training in the area of IT innovations (e.g. see the national reports from Denmark, Ireland, Lithuania, Luxembourg and Portugal among others). As the national report from Estonia summarises, insufficiently instructed users is a remarkable source of inefficiency in the Programme. The management supporting tools should be improved and stabilised and made more user-friendly. For instance, electronic applications should be improved, and the existing databases should be reviewed and renewed (e.g. including BE (nl), BE (fr), DE, DK, EL, ES, FI, LU, MT, NL, PL, PT, RO and UK). Estonia adds that the introduction of alike and often duplicating management tools needs to be analysed from the aspect of administrative burden, and the respective inefficiencies should be reduced. Slovenia points out that ICT should be more developed in at least two directions: extending the possibilities to access stored data on daily basis. Additionally, ICT should enable better utilisation of the enormous amount of data (inputted at the NA level) for monitoring and evaluative purposes. Iceland strongly recommends timely commitment of resources for the preparation of the centralised IT management system for the successor of the LLP programme. Creation of centralised Internet library or catalogue of the products created within the projects (e.g. manuals, films, photos, presentations, records, databases) was recommended by Romania. 47

48 3.4. Adequacy of resources To what extent is the level of resources/financial support adequate both in general terms and with specific focus on sub-programmes? What steps did you take to optimise the efficiency of the resources deployed for the LLP implementation in your country? What kind of rationalisation effort did you make in this respect? Overall, the Participating Countries attitudes towards the adequacy of resources have diverged. While part of the countries argue that support is quite sufficient and reasonable (e.g. BE (fr), DK, FI, IE, IS, NL, NO and UK), other countries point out that the needs strongly exceed available resources (including AT, BE (nl), CY, CZ, DE, EE, EL, IT, FR, LT, PL, PT, RO, SE and SK). However, adequacy of resources can be interpreted in different ways. For instance, it is important to know, whether the conception of adequacy takes into account the objective to promote the quality of implementation system. As the national report from Slovenia summarises, the level of financial support is adequate to support the minimum requirements of the programme and sub-programmes, but is not enough to improve the quality of the implementation. The Participating Countries argue that applications of high quality are sometimes rejected just due to insufficient resources. In addition, funding rules are artificially complicated in order to create additional hindrances for a large number of applications (e.g. see the national reports from BE (fr), BG, CY, CZ, EE, EL, ES, IT, PL, PT). Usually the lack of financial resources in sectoral programmes is covered by provision of additional co-funding from the Participating Countries budgets (including the Czech Republic, the Netherlands, Slovakia, Spain and Sweden). This apparently shows that the LLP is highly prioritised by the national decision makers. The following paragraphs concentrate on adequacy of resources in each sectoral programme. Comenius. Generally, financial support to this sectoral programme is considered as insufficient: the number of high quality applications typically exceeds available budgetary resources. (e.g. CY, CZ, FI, IT, LT and TR among other countries). As a result, the applications are rejected due to lack of finance, but not due to insufficient quality. Furthermore, the same Participating Countries argue that the size of the grants for mobility actions could be higher. Erasmus. In recent years the demand for the Erasmus programme funding has exceeded the available financial resources (e.g. in AT, CZ, DE, EE, ES, IE, LT, LU, PL, RO and TR). Usually Erasmus is co-funded from the countries budgets. Nevertheless, the reports from the Czech Republic, Estonia and Spain argue that support is too modest. The Participating Countries also argue that the funding for Erasmus should increase, since it is one of the best established actions. For example, Finland argues that more funding is needed especially for teachers and staff s mobility, intensive language courses, accommodation and travel costs. Leonardo da Vinci. Decision makers (e.g. from AT, CZ, DE, IT, LT, PT, TR and the UK) argue that the level of resources for Leonardo da Vinci is not appropriate, since the demand for financial support exceeds the budget allocated to the sub-programme. Grundtvig. The national reports from BE (fr), BG, CY, CZ, DE, EE, EL, ES, IT, LT, PL, PT, SK and the UK state that the demand for funding exceeds the supply of resources. Grundtvig receives 4% of the total LLP funding, which is spread out between large numbers of activities. The aforementioned countries argue that additional funding should be channelled to this sectoral programme, because interest among target groups is increasing every year. According to Cyprus, Learning Partnerships and In-Service Training should have a priority to get financial aid. 48

49 Finland gives its voice for additional funding for administrative costs, personnel wages and travel costs. Poland prioritises the possibility of language learning mobility for adult-learners staff. Additionally, the Flemish Community of Belgium, Germany, Italy and the Netherlands point out that more funding is particularly necessary for learning mobility projects within the whole LLP. Moreover, additional funding would also be beneficial for dissemination of the projects activities (in the Flemish Community of Belgium for Leonardo da Vinci, in Slovakia for Grundtvig, in Turkey for Grundtvig and Leonardo da Vinci) and for covering administrative costs (e.g. Denmark, Finland). In order to optimise the use of limited financial resources, the Participating Countries use the following instruments to meet the demand for funding: Jump between the lines, i.e. transfer of funds from less popular activities to the more popular ones (e.g. in BE (fr), CY, EE, ES, LT, LU, NL, PL, SK and UK). In order to detect incomplete absorption and redistribute resources, the NAs engage in continuous monitoring and evaluation of the absorption process (including BE (nl), EE, EL, NO, PL, SI and TR). The maximum aid established for the various actions by the DG EAC has been lowered and new maximum levels have been set (in Estonia, Poland and Spain among others). Merger of previously separate NAs into a single National Agency. This was done with the view of achieving economies of scale and increasing efficiency of the NAs (e.g. see the reports from Romania and the UK). Raising criteria for approval of applications (e.g. in Estonia, the French-speaking Community of Belgium and Poland). National Agency from Hungary set up a database of experts that offers a rapid and cost-saving way for attracting the needed expertise. The national Agency from Greece organises administration seminars for each of the Leonardo da Vinci actions. Summary and recommendations: To summarise, some countries argue that funding for the LLP is sufficient, while others argue that the funding is insufficient. Furthermore, the countries would like to see an increase in the level of funding for each sectoral programme this reflects the different levels of demand for different programmes in the Participating Countries. The reports provide the following recommendations for further improvements: Consideration should be given to increasing the level of flexibility in transferring financial resources between the activities and projects and sub-programmes (e.g. see the national reports from CZ, EE, EL, FI, FR, HU, NO, SE, SI, and the UK). Finish authorities recommend a more equal distribution of funds among the Participating Countries. The EU should provide additional funding for the translation of application documents and instructions, but the translation should be supervised by the NAs (report from Sweden). Simplification of decision making procedures in accordance with the budget of the actions and projects. As Sweden argues: it is not efficient in terms of resources to always require a selection committee for smaller projects. The focus on organisation and legal scutinity is geared towards large projects with substantial budgets. The majority of contracts in the LLP are for small amounts, simple activities and decision routines, reporting and checks of these should thus be at a reasonable level. Turkey proposes: Responsibility areas of institutions and criteria for eligibility of applicants should clearly be indicated, the legislation related to the Ministry of Finance should be adapted to the EU legislation, money management should be left to the universities (use of resources in the university's autonomy) and universities should be supported by the National Agency. Participants also clearly indicate that if a university does not have enough financial resources, that university should be given priority among others. 49

50 Improve funding mechanisms for Grundtvig. Sweden recommends that: the Grundtvig programme must be simplified. Having eight different sub-areas in such a small programme is simply not sustainable. It leads to substantial administrative work for few if any applicants. The report from the UK argues that adult learning should get a bigger share of the LLP budget, since with changing demographics across, European adults should be treated as an important target group. Luxembourg points out that it should be possible to transfer funds without having to make specific requests to the DG EAC. Slovakia suggests making full use of lump-sum financing of the individual mobilities under the Grundtvig and Comenius sub-programmes. As it is now impossible to move funds from one grant item to another one, it is necessary to exactly establish the costs beforehand. The funds not spent under one item shall namely forfeit and, on the contrary, overdrafts must be settled by the participant. That leads not only to an inefficient use of grants but also to withdrawals of participants from approved mobilities. Sweden summarises: Additional funds do not necessarily need to be added, but if this does not happen consideration should be given to the functions that NAs are expected to perform. 50

51 ANNEX 3. CASE STUDIES Case study template 1. Introduction 2. Case study framework The overall objective of the case studies is to assess the functioning of the Lifelong Learning Programme s sub-programmes and actions in different participant countries in order to answer the interim evaluation questions on the LLP relevance, effectiveness and efficiency of the Lifelong Learning Programme that require in-depth qualitative research. The case studied is defined as a certain action (or group of actions) of the particular sectoral subprogramme (Erasmus, Leonardo, Grundtvig, Comenius) or centralised programme (Jean- Monnet), in addition to the EU-level case studies. In total, 16 case studies were carried out. The case study analysis was carried out using: a) a semi-standardised structure; b) questions. The sources which informed the case studies were obtained through: 1. Desk research, resulting in analysis of statistical data, official documents (EU and national policy documents, calls for proposals, monitoring/synthesis reports, minutes from meetings, etc), websites and articles; 2. Semi-structured interviews with policymakers, the National Agency if applicable, institutional beneficiaries and social partners. In total, 108 interviews, 81 of them national-level, were carried out in the first case study phase, and further 8 interviews were carried out in the second phase. The expected number of interviews per case study was 8, and the result was over 7 (7,25) interviews per case study on average; 3. After conducting the interviews, experts were asked to map the flows of monitoring data from institution to the agency if applicable; identify the challenges and opportunities in the institution and the changes from the previous programming period, when the sub-programme action was a separate programme; and prepare a case study report based on a standard framework provided. This ensured that findings could be aggregated and compared across the case studies. A pilot case study on the Erasmus Students Mobility for Studies (outgoing mobility) action implementation in Lithuania was used as an example and sent to the relevant sectoral and national experts. Also, a list of documents was provided to experts by the extract of the Inception Report concerning operationalisation of relevant evaluation questions, the monitoring data on the certain decentralised actions (the National Reports), as well as an official Letter signed by the European Commission s officer in charge. Nonetheless, the experts were expected to conduct desk research independently, using various information sources. The results were summarised in case study reports, prepared according to the template presented in the next chapter. The questions were mostly similar for all the case studies, but they were also tuned to address the specificities of each case study. Below the framework (template) for the case study is provided. When preparing the case study, experts were expected ensure the compatibility of the contents of the case study monograph and the operationalisation of the sets of the evaluation questions provided in the Inception Report. 51

52 Pre-filled section Case study title Member State (if applicable) Information sources Main documents used (legal acts, national reports / implementation documents) No. of interviews and a list of interviewed organisations* Other sources (statistical sources, studies, analytical papers, etc.) *The full list of interviews (date, name of interviewee) should be presented in the Annex 1 1. INTRODUCTION (0.5pages) Case study objectives and approach Time period when case study was prepared (year and months) Main sources of data 2. RELEVANCE (1-2 pages) The main objective of this section is to describe the European Added Value of the selected subprogramme / action To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to: the national policy priorities in national-level case studies; EU policy priorities in EU-level case studies? (Evaluation question set No.1 in the Inception Report, p ) 2.2. To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the lifelong learning needs of the target groups of the programme? (Evaluation question set No.1 in the Inception Report, p ) 2.3. What is the European Added Value? Would other national schemes / instruments provide enough support to the activities funded by the evaluated LLP sub-programme / action, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? (Evaluation question set No.2 in the Inception Report, p ) 2.4. What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Additional clarification: were there preconditions for coherence of efforts? (Evaluation question set No.4 in the Inception Report, p ) 3. EFFECTIVENESS (3-5 pages) The main objective of this chapter is to provide proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the programme / sub-programme / action could be achieved Effectiveness in achieving targets and contributing to priorities: What is the progress in achieving action / sub-programme targets against financial progress of the programme? What is the extent to which the beneficiaries are satisfied with the outputs and results of different sub-programmes and actions? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? (Evaluation question set No.5 in the Inception Report, p ) What are the main external factors affecting the outputs and results of the sub-programme / action? Additional questions in specific cases: To what extent it contributes to EU education and training policies, OMC? (Evaluation question set No.5 in the Inception Report, p ) 52

53 3.2. Effectiveness in implementing horizontal policies: How does the sub-programme / action contribute to the horizontal priority of multilingualism? (Evaluation question set No.7 in the Inception Report, p ) How is the implementation of equal opportunities (EO) ensured by the action evaluated? Do centralised actions integrate the issues of EO? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? (Evaluation question set No.7 in the Inception Report, p ) 3.3. Effectiveness in integrating previous activities into the LLP: How successfully has the integration of the previous activities into the LLP been implemented? Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved (not applicable in some EU-level case studies)? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? (Evaluation question set No.9 in the Inception Report, p ) 3.4. Effectiveness in reaching the target groups and disseminating results: 4. EFFICIENCY (1-2 pages) ANNEXES How successful is the LLP in reaching the target groups and institutional beneficiaries of subprogrammes? (Evaluation question set No.10 in the Inception Report, p ) To what extent are the Programme results adequately disseminated and exploited at both horizontal and vertical levels? (Evaluation question set No.11 in the Inception Report, p ) 4.1. Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? (Evaluation question set No.13 in the Inception Report, p ) 4.2. Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? (Evaluation question set No.13 in the Inception Report, p ) Is the monitoring information used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? (Evaluation question set No.13 in the Inception Report, p. 78) Table 5.2. List of interviewees No. Institution Type National agency / national authorities / institutional beneficiary / social partner / other Name, surname and position of interviewee Date and type of the interview Face to face / telephone, etc. 3 The last two questions concerting efficiency were modified in some case studies. E.g. in the case study on Transversal KA2 Languages, question 4.2 was formulated as To what extent is the implementation and management structure of the KA2 and other relevant actions efficient and well functioning i.e. funds reach the intended target groups; are disbursed with no delays; with no systemic irregularities? Could the KA2 be effectively decentralised with better reach of target groups less administrative deficiencies and no loss of European visibility and dimension (and vice versa)?, while question 4.3 was formulated as How effective is the cooperation between the different management bodies (the Commission the Executive Agency the National Agencies national authorities the LLP Committee)? To what extent does the Commission fulfil its guiding role in the process? 53

54 3. Interview questionnaires 3.1. Interview questionnaire for national institutions and national social partners This questionnaire covers the extended list of questions for the interviews with national authorities and agencies as well as the national social partners, to be conducted while preparing case studies on decentralised actions of the Lifelong Learning Programme (Comenius, Erasmus, Leonardo da Vinci and Grundtvig), also for the case studies on Jean Monnet Excellence Centres and Chairs and Jean Monnet Operating Grants to Specified Institutions. For each of the interviews specific questions need to be selected from this questionnaire, matching the expertise of the interviewee(s). Detailed questions may be added to expand the broader groups of main questions. 1. Questions related to relevance: Pertinence of objectives 1.1. To what extent the intermediate and specific objectives of the <evaluated action / sub-programme> are integrated / coincide with the national policy priorities? To what extent the objectives of the <evaluated action / sub-programme> are relevant to the national priorities in your country? Could you identify such cases where national and EU-level objectives diverge? (where applicable) You may want to provide a list of objectives of the <evaluated action / sub-programme> 1.2. Are there / were there any preconditions installed to ensure coherence of efforts for EU-level and national policy cooperation and coordination? Can there be any good practice examples identified? What is the mechanism for ensuring that the priorities for the calls for proposals correspond to the shifting national policy priorities and socio-economic needs? What improvement could be recommended in this area for the period, post-2014 period? European added value 1.3. If, hypothetically, there was no <evaluated action / sub-programme>, would other national schemes/ instruments provide enough support to the activities supported by this action (e.g. students mobility, school partnerships etc.)? What would be the likely level of funding and the likely level of outputs achieved? What are the most important alternative national schemes or instruments? What is the additional value created by the <evaluated action / sub-programme> in terms of providing support to the education and training system? 1.4. If, hypothetically, the LLP would not have been established, what would be the level and intensity of the following activities (see below)? How does the European dimension of these sub-programmes contribute to the effectiveness of cooperation? LLP integration a) Meetings, discussions, etc. between national authorities of Participating Countries; b) Policy cooperation, learning and interchange between the Participating Countries c) Identification of common European-wide objectives; d) Development of common EU-wide policy tools (indicators, databases of good practice, tec.); e) Development of common EU-wide tools and methods for the recognition and evaluation of competences and skills; f) EU-level studies and research concerning lifelong learning issues; g) Introduction of European dimension in the national systems of education and training; h) Policy reforms aiming at increased accessibility of lifelong learning opportunities to the disadvantaged citizens; i) Funding for mobility of individuals provided by national, regional or local authorities The Lifelong Learning Programme integrated different instruments into a single programme. Are there any synergies of implementing all elements of the Programme jointly? Could you identify any outputs or results 54

55 that have been achieved (to a larger extent) due to joint implementation of the sub-programmes? What evidence do you have for this? 1.6. Is there any evidence of duplication (e.g. between centralised and decentralised actions or between different sub-programmes of the Lifelong Learning Programme)? If yes, could you provide some comments, examples? What improvement could be recommended in this area? 2. Questions related to effectiveness: Outputs and results achieved 2.1. How the quality and accessibility of your national lifelong training system has improved as a result of the <evaluated action / sub-programme> (according to the list below)? a) Contributed to improving the quality, attractiveness and accessibility of the opportunities for lifelong learning; b) Contributed to reinforcing the contribution of lifelong learning to social cohesion, active citizenship, intercultural dialogue, gender equality and personal fulfilment; c) Contributed to increasing participation in lifelong learning by people of all ages, including those with special needs and disadvantaged groups, regardless of their socio-economic background; d) Contributed to promoting language learning and linguistic diversity; e) Contributed to development of quality assurance in all sectors of education and training. Do you have any evidence for this, or could you point to a relevant research on this topic? 2.2. How successful is the <evaluated action / sub-programme> in achieving the planned outputs and results (for the period) in your country? Do you measure the satisfaction of the beneficiaries? Are the results available? 2.3. In your opinion, what explains progress in achieving the objectives and targets of the <evaluated action / subprogramme > as well as the extent to which the beneficiaries are satisfied with the outputs and results, for example: a) Ambitious objectives and targets of the <evaluated action / sub-programme>; b) The LLP budget and its breakdown according to the sub-programmes and the Participating Countries; c) Specific needs of the beneficiaries; d) Administrative burden; e) Other factors? 2.4. According to your experience, what external factors have affected the outputs and results of the <evaluated action / sub-programme>, for example: a) Particularities of national education and training systems; b) Economic factors; c) National education and training policies; d) Lack of support from main socio-economic partners; e) Lack of capacity and willingness of (potential) beneficiaries; f) Lack of facilitating mechanisms for mobility (mobility barriers); g) Values and attitudes of individuals; h) Lack of financial and human resources; i) Lack of charters and certificates (as minimum requirements for participating in the Programme); j) Other factors (please provide comments, examples)? What has been the influence of these factors (strong/weak; positive/negative)? 2.5. What external factors are likely to affect the <evaluated action / sub-programme> implementation in the future (the period; post-2014 periods)? 55

56 Horizontal policies 2.6. How does the <evaluated action / sub-programme> contribute to the horizontal priority of multilingualism? Is the issue of multilingualism prioritised in the Calls for Proposals? To what extent the target groups (reluctant learners, people with little or no experience of learning) are participating in the funded projects? Do you have any evidence for this? 2.7. In your opinion, which of the two following approaches are more successful in contributing to the horizontal policy of multilingualism: a) Differentiated, when policy is implemented through a separate sub-programme such as the KA2 of the Transversal programme; b) Integrated, when policy is implemented through priorities set in separate sub-programmes or annual Call for Proposals? What evidence exists of synergies between two approaches? Do you see any duplication? 2.8. How does the <evaluated action / sub-programme> contribute to the horizontal policy of equal opportunities (equality between men and women, non-discrimination as well as inclusion of learners with special needs)? Is the issue of equal opportunities prioritised in the Calls for Proposals? What steps could be recommended to increase the contribution? In your opinion, is the integrated approach sufficient or additional differentiated actions (e.g. a separate transversal sub-programme) could be recommended? 2.9. Have you organised any events at the national level promoting the following horizontal issues: multilingualism, equal opportunities, for example, related to the issues of equality between men and women, nondiscrimination as well as learners with special needs? (where applicable) Do you have any evidence for this, or could you point to a relevant research on this topic? Integration of previous activities into LLP How would you evaluate the effects of integrating the previous activities into the Lifelong Learning Programme, for example: a) Integration allowed supporting new actions as well as activities that would not have been otherwise possible; b) Integration allowed better mainstreaming of the horizontal policies in the sub-programmes and actions that would not have been otherwise possible; c) Integration allowed linking the sub-programmes and actions, introducing cross-cutting actions that would not have been otherwise possible; d) Integration allowed reaching new the target groups / audiences of the Programme; e) Integration allowed reaching more representatives of the target groups / audiences of the Programme; f) Integration allowed harmonisation in the procedures of the Programme; g) Integration difficulties (financial, organisational or procedural) have been overcome during the Programme implementation; h) Duplication of efforts has been reduced as a result of integration; i) There was administrative simplification as a result of integration Did any difficulties occur while integrating the Programme and how were these solved? In what ways further integration of the LLP could contribute to the utility of the programme? Reaching the target groups Have there been any difficulties in reaching the target groups of the <evaluated action / sub-programme>? What were the reasons behind these difficulties? How could these difficulties be overcome? What improvement could be recommended in this area for the period, post-2014 periods? Dissemination and exploitation of results How are / were the results of the <evaluated action / sub-programme> disseminated? Is there an overarching dissemination strategy at the national / sub-programme level? Are there any efforts for mainstreaming 56

57 horizontally (among other education and training providers in your country or other countries) the good practices and project results? What kind of information is generally disseminated (concrete products, process experiences, new ideas). What importance do stakeholders (project holders; national agencies) attach to the process of disseminating the project results? Is dissemination an integral part of project activities? Are there any bottlenecks in disseminating project results (human, financial and organizational resources; project / product related; contextual)? Are you aware of any good practices being adopted and implemented by other (national and international) stakeholders as a result of dissemination measures? Are you aware of any good practices and project / sub-programme results that have been mainstreamed vertically into national policy? Do you have any evidence for this, or could you point to a relevant research on this topic? In your opinion and based on your expertise, what actions could be taken to increase the exploitation of Lifelong Learning Programme and its sub-programmes results? Have the objectives of the Transversal programme and Jean Monnet been reflected in these questions e.g. capacity building, awareness of European dimension etc.? What preconditions are installed to increase sustainability of projects? Does the selection process of new projects or the monitoring system emphasise the sustainability and longer term impact of projects? For example, is evidence of sustainability of previous projects requested when previous applicants apply for the next project? What are the main constraints affecting the dissemination of the results of the projects and other experiences as well as their adoption, for example: a) Insufficient quality of the project results and other experiences; b) It is difficult to reach appropriate target groups; c) Insufficient budget for using more or other dissemination instruments; d) Not enough expertise for dissemination; e) Resistance to change by certain target groups; f) More innovative results and practices are being disseminated; g) Good practices do not fit with the prevailing values or regulations; h) Other constraints? 3. Questions related to efficiency: Clarity and stability of the overall legal / formal framework of programme management and control system 3.1. The programme management and control system is extensively described and regulated by Guide to NAs, Commission-NA agreement and other LLP management documents. Are there any areas where requirements for management, control and monitoring are not clear / sufficient? That is, you would say it is not entirely clear how a certain function would have to be performed or / and why it is required. When providing examples, please be specific about: a) the programme and the type of projects; b) the phase of project cycle or specific function (such as application, assessment, financial supervision and control, counselling, reporting and similar); c) reasons why a certain area is not clear (e.g., it is a new area / requirement; it is an old area / requirement, which recently changed; there are differences between your and the Commission s interpretation of any requirement / function; etc.); d) what action could eliminate the gap (e.g., revision / improvement of legal provisions; revision/improvement of standard document / template; better guidance / explanations / training from the Commission; etc.). Adequacy of management resources 3.2. In the Yearly NA Reports there is a clear trend to complain about constant work overload. a) In general, would you say you possess adequate resources to ensure sound management of the programme? b) What are they key areas of shortages, if any? That is, please provide examples of areas/functions, which you do perform adequately, but you feel they are not worth that (i.e., the area requires a lot of administrative 57

58 resources and effort, but generates little added value), and/or which you have problems to perform adequately (i.e., shortage of resources results in being late, performing not up to a standard, etc.). When providing examples, please be specific about: 1) the programme and the type of projects; 2) the phase of project cycle or specific function (such as application, assessment, financial supervision and control, counselling, reporting and similar). c) Do you use external assistance (external experts, contractors) to perform some of the functions related to the management of LLP and in particular the <evaluated action / sub-programme >? If yes, in which areas (e.g., during / for counselling (provision of advice and guidance to applicants and beneficiaries), assessment, monitoring (incl. monitoring meetings), checks (e.g., on-the-spot checks)? Use of monitoring information 3.3. A lot of monitoring information is being generated by beneficiaries (through their monitoring reports). Do you use it for management purposes? What would be other examples of its use besides putting it into Yearly NA reports to National Authority and European Commission? Please be specific when providing your answers Interview questionnaire for the institutional beneficiaries This questionnaire covers the long list of questions for the interviews with EU-level or national institutional beneficiaries, to be conducted while preparing case studies on centralised or decentralised actions of the Lifelong Learning Programme. For each of the interviews specific questions need to be selected from this questionnaire, matching the expertise of the interviewee(s). Detailed questions may be added to expand the broader groups of main questions. 1. Relevance of the Lifelong Learning Programme: 1.1. Would your project / activity have taken place without funding from the <evaluated action / sub-programme>? 1.2. Are there any national sources for the same type of activity as your project that was funded by the <evaluated action / sub-programme>? 1.3. Did participation in <evaluated action / sub-programme> meet initial objectives of your organisation? 1.4. Would other alternative types of support be more efficient in addressing the same objectives? If yes, what alternative types of support / actions would that be? 2. Effectiveness: Achieving outputs and results 2.1. Were the planned outputs and results of your project achieved as expected? Did you encounter any significant obstacles? Did the achieved outputs and results match your institution s needs and the needs of the target groups (e.g. students, teachers)? 2.2. Do you have any evidence that participation in your project had any influence on the actual or possible career benefits for the participants, for example finding a job, receiving promotion, financial or non-financial benefits? If yes, please provide examples and comments What influence participation in the <evaluated action / sub-programme> had on your organisation (if any), for example: All programmes: - Language proficiency; - Intercultural competences; - Professional, specialist know-how; - Changes of attitudes and perceptions regarding the host country; - Changes of attitudes and perceptions regarding European integration; 58

59 - Professional achievements; - Team skills; - Interpersonal skills; - Self assurance / self-confidence; In Erasmus: - Academic knowledge, specific know-how and practical skills of the staff; - Better expertise and experience on pedagogical methods; - Better understanding of the higher education system in the host country; - Establishing or enlarging the network of partners or contacts; - Transnational cooperation in teaching content and practice; - Broadening and deepening the range and content of study programmes, courses or teaching modules; - Improving performance of higher education institutions through specific strategies or measures/actions; - Fostering practices of quality assurance; - Improvement of international profile; - Creating links between higher education institutions and enterprises, private sector; - Greater awareness of European dimension of higher education; - Development / mainstreaming of ICT-based content and practice; - Other important influences (please specify)? In Leonardo da Vinci: - Greater awareness of European dimension of VET; - New contacts within the VET field; - New knowledge over different educational methods, practices, materials; - Awareness of VET system in host country; - Awareness of pedagogical approach in host country; - New knowledge over ICT related learning/schooling methods; - Ability to design work processes; - Found (better) work placement; - Professional advancement; - Attained a higher income; - Achieved greater responsibility in the workplace; - Other important influences (please specify)? In Grundtvig - Greater awareness of European dimension of adult education; - Greater awareness of other actors in the field of adult education in general; - New contacts within the field of adult education; - New knowledge over different educational methods, practices, materials; - Awareness of adult education system in host country; - Awareness of pedagogical approach in host country; - New knowledge over ICT related learning/schooling methods; - Other important influences (please specify)? In Comenius: - Improved practice in teaching and learning; - Development of a European and / or wider global dimension to pupils learning; - Sharing of educational good practice at a European level; - Strengthening of educational partnerships and networks; - Strengthening of the quality of in-service training and professional development for serving teachers; - Strengthening of the quality of initial training for intending and initial teachers; - Improved language skills among i) teachers and ii) pupils; - Improved access to information about European and wider global educational partnership opportunities; - Improved European links and interaction between Local Authorities, schools, school education organisations, business and industry (Comenius REGIO and/or other Comenius actions); - Other important influences (please specify)? Do you have any evidence on the positive or negative influence? If so, could you provide comments, examples? 59

60 2.4. How would you assess your overall satisfaction with participation in the <evaluated action / sub-programme>? What were the main factors positively affecting the outputs and results of the project, for example: a) Factors external to the action sub-programme: - Mobility barriers (e.g. the recognition of skills, different living standards between countries, insurance policies, legal obstacles, etc.); - Language barrier; - Changes in the financial position of the institution / individual (e.g. owing to economic downturn); - Increasing variety of learners backgrounds (regarding educational level / social origin / family background / social class, earned income, gender, age, employment status, ethnic origins, urban / rural, etc.); - Support / lack of support from socio-economic partners; - Capacity / willingness of potential beneficiaries; - Values and attitudes of individuals; - Changes in national education and training policy (e.g. resulting in other increased demands on staff, i.e. increased business of staff); - Other factors? b) Content of the sub-programme / action: - Objectives; - Priorities; - Target groups; - Actions; - Financial provisions? c) Management of the action / sub-programme: - Clarity of requirements for applicants, application procedure; - Clarity and transparency of selection criteria; - Timeliness / duration of selection / award process; - Clarity of expenditure eligibility requirements; - Clarity of procedures pertaining to financial management; - Timeliness / duration of payment processing procedures; - Clarity of project reporting requirements; - Other practicalities (please specify)? What actions would you recommend for the future participants or <evaluated action / sub-programme> administrators that could help decrease the influence of negative factors, or increase the influence of positive factors in the future? Integration of previous activities into the LLP 2.5. Are you aware that the <evaluated action / sub-programme> is a part of a larger EU Lifelong Learning Programme (LLP) including other sub-programmes (such as Erasmus, Grundtvig, Comenius, Transversal, Jean Monnet)? If yes, did integration of the different sub-programmes into single Programme cause additional outputs for you and other organizations that would not have otherwise been achieved? Please provide examples. Reaching the target groups 2.6. To what extent your project/activity has reached (is likely to reach) the target groups of your project? Do you have any evidence on this? Did your project/activity involve persons with special educational needs (e.g. disabled persons), socially and / or economically disadvantaged groups, people excluded from the education and training systems (e.g. early school leavers, those with little or no education), people seeking to re-enter the labour market, etc.? If yes, how successfully did you manage to reach those groups? What were the major obstacles? Dissemination and exploitation of results 2.7 How important do you consider the process of disseminating the project results? How successful has the implementation of dissemination / exploitation plan (if there is one) of your project been (or is likely to be in 60

61 future)? What kind of information is generally disseminated (concrete products, process experiences, new ideas)? 2.8 Has your organisation applied (is likely to apply) the following instruments in order to disseminate the project results and experiences gained from your participation in the <evaluated action / sub-programme>: a) Internet websites; b) Manuals, readers, other material; c) Handbooks, books; d) Expositions, exhibitions; e) Studies, reports, other publications; f) Photographs; g) CD-ROM; h) Seminars, conferences, workshops, other events; i) Networks of experts; j) Video/internet conferencing; k) Round tables; l) Newsletter; m) TV; n) Radio; o) Newspaper; p) Other? 2.9. What are the target groups for the dissemination of your project results and experiences, for example: a) Education providers in my area of education (e.g. adult education); b) Education providers in another area of education; c) Associations; d) Enterprises; e) Public authorities; f) Public service providers; g) Counselling bodies; h) Research centres; i) NGOs; j) Other? Is there any evidence that the results of your project / activity have been exploited after the completion of the project / activity? If yes, please provide examples (for example, of transfer and commercialisation). For example, did your dissemination and exploitation activities lead (or are likely to lead in the future) to the following: a) Increased awareness of quality, relevance and effectiveness of the results of your project by target groups of dissemination and exploitation activities? b) Adoption and / or application of the results of your project by target groups of dissemination and exploitation activities? c) Transfer of results of your project to appropriate decision-makers at local, regional, national or European levels? Will your project be continued? Will results of your project be sustainable and accessible after the end of the project? Based on your experience, what are the main constraints affecting the dissemination of the project results and other experiences as well as their adoption, for example: a) Insufficient quality of the project results and other experiences; b) Lack of dissemination strategy / activities (dissemination is not part of the project); c) It is difficult to reach appropriate target groups; d) Insufficient budget for using more or other dissemination instruments; e) Not enough expertise for dissemination (for example, insufficient staff expertise for virtual dissemination); f) Lack of motivation / interest / time in dissemination / exploitation within organisation; g) Resistance to change by certain target groups; h) More innovative results and practices are being disseminated; i) Disseminated practices are not perceived as better than other practices disseminated at the moment; 61

62 j) Disseminated practices are not perceived as better than the current situation or products k) Good practices do not fit with the prevailing values or regulations; l) Other? What actions (for example, by institutional beneficiaries or the sub-programme managers) could be taken to encourage better dissemination of projects results in the future? Horizontal policies Has your project been successful or is likely to be successful in addressing any of the following issues of equal opportunities: a) Gender; b) Racial/ethnic origin; c) Roma; d) Religion/belief; e) Disability/special needs; f) Sexual orientation; g) Age? Do you have any evidence on this? What is the distribution of the beneficiaries who participated in your project/activity (e.g. according to gender, etc.)? What are the problems and obstacles in addressing the following issues of equal opportunities, for example: a) It is difficult to reach the target groups to whom the issues of equal opportunities would be relevant; b) Insufficient information available as to how the issues of equal opportunities could be addressed in the project applications and their implementation; c) The issues of equal opportunities are not prioritised enough in the Calls for Proposals of the LLP; d) The issues of equal opportunities are not prioritised enough in national policy; e) Addressing the issues of equal opportunities requires additional financial resources; f) The supervision of the beneficiaries does not sufficiently concern the issues of equal opportunities; g) Other problems and obstacles? How could these problems and obstacles be overcome in the future, e.g. with the programme administrators support? How likely it is that your organisation would have sought to address the issue of equal opportunities in ensuring access to lifelong learning, if your project would not have been supported by the LLP? 3. Efficiency: Clarity and stability of the requirements pertaining to LLP grant management and reporting 3.1. Are there any areas where requirements for grant management (both financial and administrative aspects) and reporting are not clear / sufficient? That is, you would say it is not entirely clear how a certain function would have to be performed or / and why it is required. When providing examples, please be specific about: a) The phase of project cycle or specific function (such as application, assessment, financial management, control from NA, reporting and similar); b) Reasons why a certain area is not clear (e.g., it is a new area / requirement; it is an old area/requirement, which recently changed; there are differences between you and NA in interpretation of a requirement / function; etc.); c) What action could eliminate the gap (e.g., revision / improvement of legal provisions; revision/improvement of standard document / template; better guidance / explanations / training from the NA; etc.). Adequacy of management resources 3.2. In the reports there is a clear trend to complain about excessive administrative burden and constant work overload. A. In general, would you say you possess adequate resources to ensure sound management of the grant? 62

63 B. What are they key areas of shortages, if any? That is, please provide examples of areas / functions: - Which you do perform adequately, but you feel they are not worth that (i.e., the area requires a lot of administrative resources and effort, but generates little added value), and / or - Which you have problems to perform adequately (i.e., shortage of resources results in being late, performing not up to a standard, etc.). When providing examples, please be specific about the phase of project cycle or specific function (such as application, assessment, financial supervision and control, counselling, reporting and similar). 63

64 EU-level case studies 1. Programme-Policy Link 1. INTRODUCTION Since national education and training policies are the responsibility of the Member States, the EU could only supplement and support Member States actions. According to the decision establishing the LLP, the Programme aims to foster interchange, cooperation and mobility between education and training systems within the Community so that they become a world quality reference. 4 The intervention logic of the Programme, which was presented in the Inception Report, identified the system-level specific objectives of policy-learning, change of national systems, and the development of European dimension of lifelong learning. As stated in the Inception Report, this EU-level case study contributes to outlining the links between the LLP, the cooperation at the EU level, as well as national policies in the area of education and training affected by the Open Method of Coordination (OMC). This case study is based on the following sources of information: - desk research of the policy and programme documents; - three EU-level interviews; - data of the survey of the Programme beneficiaries; - data of the survey of the National Authorities and Agencies; - monitoring and evaluation information; - analysis of the National Reports. Mechanisms of the OMC influence The national systems of lifelong learning in the Member States are affected by the OMC through national policies in the area of education and training. The OMC is defined as an intergovernmental method of soft coordination by which the Member States are evaluated by one another, with the Commission's role being one of supervision. Two main processes of the OMC are relevant in the area of lifelong learning: - The EU Lisbon strategy, to be replaced by the new EU 2020 strategy; - The process of Education and Training 2010 replaced by the new Strategic Framework According to the Inception Report, the renewed Social Agenda, which is linked to the new Strategic Framework 2020, is not considered under this evaluation question. It is important to draw upon the OMC literature in explaining how changes in the reform agendas or (substantive and procedural) policies could occur at the national level. One OMC impact assessment identified five main mechanisms of the OMC influence 5 : - external pressure, which is often associated with benchmarking; - financial support from the EU, including the Structural Funds; - socialisation and discursive diffusion; - mutual learning; and - creative appropriation by domestic actors or, in other words, strategic use of the OMC on the national level. Under the OMC of Education and Training the most relevant mechanisms of influence appear to be the following: - European benchmarking and evidence-based policy making. European benchmarks were set both under the Education and Training 2010 work programme as well as the Strategic Framework 2020 in this area. They invite the Member States to consider how and to what extent they could contribute 4 Decision of the European Parliament and of the Council establishing an action programme in the field of lifelong learning, Decision No. 1720/2006/EC, 15 November, 2006, Official Journal of the European Union, L327 of , p J. Zeitlin. The Open Method of Coordination and National Social and Employment Policy Reforms: Influences, Mechanisms, Effects, in Heidenreich, Martin, and Zeitlin, Jonathan (eds.), Changing European Welfare and Employment Regimes: The Influence of the Open Method of Coordination on National Reforms. Routledge/EUI Studies in the Political Economy of Welfare, London: Routledge,

65 to the collective achievement of the European benchmarks through national actions. Also, in order to ensure the use of evidence in policy making in the area of education and training, it is important to develop and apply studies, analysis and other forms of evidence in the OMC process; - Peer learning activities and other fora of learning (such as networks, partnerships, clusters, expert groups, conferences and seminars, panels, web-based cooperation). The main forum of learning is peer learning, which is defined as a process of cooperation at European level whereby policy makers and practitioners from one country learn, through direct contact and practical cooperation, from experiences of their counterparts elsewhere in Europe in implementing reforms in areas of shared interest and concern. 6 Figure The Open Method of Coordination: its influence mechanisms and outcomes. Mechanism of the OMC Open Method of Coordination Outcomes of the OMC European benchmarking and evidencebased policy making Peer learning activities and other fora of learning Lisbon Strategy/2020 Strategy Education and Training 2010/2020 Strategic Framework National reform agendas Substantive and procedural policy changes at the national level These two mechanisms of the OMC influence are inter-connected: to identify the needs and challenges and observe progress of the Member States, it is important to have solid empirical evidence, which could be discussed in various fora of learning (Figure 3.0.1). Outside these mechanisms, the OMC can influence the willingness of the Member States to pursue national reforms through raising political ambition. All these mechanisms of the OMC influence should contribute to changing the reform agendas and, eventually, actual education and training policies at the national level, thus affecting the national systems of lifelong learning and their providers. 2. RELEVANCE Main achievements and problems of the OMC process in education and training In its impact assessment the European Commission recognised that the majority of countries, social partners and stakeholders acknowledge that the OMC in education and training has effectively supported the development of national education and training policies. Particularly the OMC has provided a knowledge base for national policy making and objective setting, achieved greater convergence between national policies, provided comparable data, established benchmarks and indicators, and brought policy makers together to discuss policies, issues and good practice. 7 There is other evaluation evidence that peer learning activities, which constitutes the main fora of learning, are informing national policy or management decisions. For instance, it was found that in one work group of Education and Training 2010 Estonia learned from the Irish practice of using the EU structural funds to foster lifelong learning, and Greece was learning from the UK in the same area. 8 Nevertheless, despite a positive assessment of the OMC in education and training, there are some problems in this process. First, ownership and visibility of the OMC objectives is observed to be low at the national 6 The European Commission. Progress towards the Lisbon objectives in education and training: Analysis of implementation at the European and national levels. Commission staff working paper, Brussels, 2009, p Commission staff working document accompanying the communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, An updated strategic framework for European cooperation in education and training. Impact Assessment, Brussels, SEC(2008) 3047, p European Commission. Evaluation of the Integrated Guideline Package for Growth and Jobs. Final Report, 2008, p

66 level. Second, there was insufficient involvement of various stakeholders in the OMC process. 9 Therefore, the European Commission called to strengthen the focus, political commitment and visibility of the OMC process as well as strengthen the involvement of stakeholders in this process. The importance of good dissemination of evidence as well as wide participation of the stakeholders during peer learning in the OMC process is also emphasised in the academic literature. 10 New strategic framework for education and training The Programme is relevant to the OMC process because it can address both the existing and future needs of European cooperation at the EU level. Therefore it is necessary to take into consideration the agreed and proposed changes to the EU strategies as well as the OMC process in the area of education and training. On 12 May 2009 the Council adopted a strategic framework for European cooperation in education and training 11, replacing the Education and Training 2010 work programme. One of the most important changes is limiting the Strategic Framework Education and Training 2020 to only four strategic objectives (Making lifelong learning and mobility a reality; Improving the quality and efficiency of education and training; Promoting equity, social cohesion and active citizenship; Enhancing creativity and innovation, including entrepreneurship, at all levels of education and training) compared with 13 objectives of the 2010 work programme. Starting from 2009, a number of short-term priorities will be set for three-year cycles, making any necessary adjustments faster (see comparison in Table 3.0.1). The interview programme supports the finding that the new structure of the Education and Training 2020 should allow for a more flexible implementation compared with the initial structure of the 2010 work programme. Table Cycles of the Lisbon strategy (the Education and Training 2010 work programme) and the EU 2020 strategy (the Strategic Framework 2020 in education and training) No. Criterion The Lisbon strategy The EU 2020 strategy EU-level targets in the area of education and training No targets 2 headline targets set and to be quantified National targets in the area of education Not mandatory To be set by the Member States and training National reform programmes Prepared To be prepared 4. Monitoring of the implementation of the strategy Annual, at the Council level Annual, at the Council and European Council level Education and Training 2010 work programme Strategic Framework 2020 in education and training 1. Number of objectives Number of benchmarks 5 5 (to be increased) 3. Monitoring of indicators and Annual Annual benchmarks 4 Reporting Bi-annual Every three years (from 2012) Also, political ambition of the EU education and training policy will be raised in the EU s new Strategy for Jobs and Growth, replacing the Lisbon strategy. The European Council agreed main elements of the EU 2020 strategy, which is consistent with the Strategic Framework Education and Training One of the main elements of this strategy is the headline targets that will guide its implementation and monitoring. The targets of education levels (in particular aiming to reduce early school leaving rates and increase the share of the population with tertiary or equivalent education) and social inclusion (in particular the reduction of poverty) will be quantified at its meeting of June On the basis of these targets, the Member States will set their own targets, taking into consideration their relevant starting positions and national circumstances. Also, similar to the Lisbon strategy cycle, the Member States will draw up their national reform programmes setting out the actions to implement the new EU strategy. 9 Commission staff working document accompanying the communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, An updated strategic framework for European cooperation in education and training. Impact Assessment, Brussels, SEC(2008) Bernard H. Casey and Michael Gold. Peer review of labour market programmes in the European Union: what can countries really learn from one another? Journal of European Public Policy 12:1 February 2005: Council conclusions of 12 May 2009 on a strategic framework for European cooperation in education and training. 12 The European Council. Conclusions of 25/26 March Brussels, 26 March 2010, p

67 More specifically, peer learning should be streamlined and organised in a more flexible way in order to adapt to the changing needs of political priorities and specific themes. 13 Since 2005 peer learning focused on seven clusters of thematic activities. One interviewee argued that in view of improving the effectiveness of peer learning activities joint preparation by the European Commission and the Member States as well as higherlevel representation in peer learning and wider dissemination of peer learning lessons are important. Overall, the EU 2020 Strategic Framework in Education and Training will raise political ambition of the EU policy and should enable more flexible implementation of the OMC. However, there is a need to assess how the LLP is contributing to the OMC process and how its contribution could be improved in the future. 3. EFFECTIVENESS Contribution of the Programme to EU policy According to the LLP Activity Report, in the period the Programme made contributions to the progress of the cooperation between Member States, notably through peer learning activities and through projects contributing to the European Qualification Framework. 14 The National Reports also indicate a contribution of the Programme to the achievement of the EU priorities set out in Education and Training For instance, according to the UK national report, the LLP has encouraged countries to work together at the European level to share best practice and case studies to make progress towards the priorities. 15 However, it is important to assess how the Programme is contributing to the EU policy and through which sub-programmes/actions. This assessment is based on the logical model presented in Figure 3.0.2, linking the Programme outputs to its impacts. It is important to recognise that that there is no linear relationship between evidence (a main output of the Programme) and policy decisions (an impact of the Programme) because different types of activities are informing each other. Therefore, a more interactive approach, focusing on links and synergy effects among various stakeholders of the policy process, is required to explain the Programme-policy link. According to the logical model, the Programme can contribute to the political decisions concerning the EU policy through raising the competences of main stakeholders as well as facilitating their consensus in certain areas of education and training policy. Figure Logical model of the Programme-policy link Source: prepared by the authors. Policy cooperation in the area of education and training was prompted by a consensus of the Member States on the EU Lisbon agenda. The interview programme shows that there is a consensus on the role of education and training (its basic principles and objectives) in the new EU strategic documents. Among other 13 Commission staff working document accompanying the communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, An updated strategic framework for European cooperation in education and training. Impact Assessment, Brussels, SEC(2008) Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p Interim Evaluation of the Lifelong Learning and Youth in Action Programmes United Kingdom Response, 28 May 2010, p

68 factors, the Programme has contributed to reaching this consensus among the policy makers and other stakeholders. However, the interviews also show that the design of specific policy instruments and other issues of policy implementation are shared by the stakeholders to a lesser extent, owing to different situations in the Member States. Therefore there is a need to make evidence and learning more specific and thematic in the OMC process. According to the survey of the LLP beneficiaries, about 37% of the KA1 respondents and 34% of the KA2-4 respondents agree that their project strongly contributed to raising the competences of policy makers and education/training specialists on the new demands of society and knowledge about good practice examples. It illustrates that the Transversal programme is successfully contributing to the EU policy by raising the competences of the stakeholders. The centralised actions of the KA1 are better contributing to informing policy making (33,3% of the respondents strongly agree with this statement) compared with the decentralised actions (the study visits) (only 17,7% strongly agree). Contribution of the Transversal programme The Programme contributes to the mechanisms of the OMC influence through the Transversal programme on the top-down basis. KA1 of the Transversal programme supports study visits, studies and comparative research as well as policy cooperation. The survey of the beneficiaries shows that the exchange of information, experience and good practice is the main activity and output of KA1 (according to about 17% of the respondents). The objectives of the Action Studies and Comparative research are to support the Education and Training 2010 process through studies and comparative research in education and training fields at European level; to contribute to evidence-based policy-making by producing state-of-the-art scientific knowledge on specific sectors of education and training; to promote the creation of research consortia and cooperation between European research institutes and researchers in the field. The Action became more targeted in the period, focusing on specific priorities proposed in the annual calls for proposals on the basis of the Education and Training 2010 work programme. The analysis of the monitoring data indicate that the majority of the projects are complementary with the Education and Training 2010 work programme: 57% of all projects in 2008, 67% in 2009 (Table 3.0.2; no 2007 monitoring data are available). Table Complementarity of studies and comparative research with the Education and Training 2010 work programme and other policies The Education and Training 2010 work programme Other policies (per one policy) applications approved 57% 17% applications approved 67% 17% Source: analysis of the monitoring data provided by the Executive Agency. The introduction of dissemination plans allowed improving the dissemination of project results under this action. Also, it enabled better combining scientific evidence with its dissemination tools (e.g. presentation of studies at international conferences and other events). However, according to the interview programme, this action is considered not very useful for informing policy making because of an implementation gap between the specific nature of its action and the quality of project proposals as well as their results. Also, under the action of policy cooperation KA1 supports studies and other procurement procedures (the framework contract), provides grants to a number of instruments and bodies supporting mobility and lifelong learning (NARIC, Eurydice, Europass, Euroguidance), specific calls for proposals as well as Presidency events (see Table below for the breakdown of the budget according to each centralised action of KA1). 16 Table Centralised actions of the KA1 of the Transversal programme: the breakdown of the budget in 2007 and 2008, No. Title of action Total 1. Policy cooperation Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p

69 No. Title of action Total 1.1. Centralised Calls for Proposals Studies and other procurement procedures Other grants to national bodies (NARIC, Eurydice, Europass, Euroguidance) 1.4. Presidency grants and other monopolies Studies and comparative research Source: Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p The interview programme shows that the procedure of framework contract is very useful to the mechanisms of the OMC. This action directly contributes to policy making by providing evidence as well as supporting the exchanges between the Member States and the European Commission. For instance, under the framework contract back-up consultancy services are provided for the implementation of the Education and Training 2010 work programme, including consultancy services for peer learning activities, the development of a website for the dissemination of best practices, a comparative analysis of the progress made in adopting national lifelong learning strategies. 17 This indicates the synergy of evidence and learning fora under this action. Specific calls for proposals address the problems of the OMC identified in the Commission s impact assessment. 18 For instance, the problem of ownership and awareness in the OMC process is addressed through the specific call for proposals Raising awareness of lifelong learning Education and Training Its purpose was to raise the profile of European cooperation in education and training and national lifelong learning strategies through national awareness-raising. Under this call for proposals 17 projects were funded in 2007, 11 projects in According to the LLP Activity Report, the National Authorities showed a strong commitment to the development and implementation of lifelong strategies. 19 Also, a specific call for proposals Projects aiming to implement and develop the EQF was announced to support the development and preparing the implementation of this framework. This is one of the main cooperation initiatives at the EU level supported under KA1 of the Programme. Eleven EQF projects were supported under the 2007 call. The EU-level case studies of the LLP interim evaluation provide some evidence that other actions of the Transversal programme contribute to the EU priorities in the area of education and training. For instance, KA2 of the Transversal programme is contributing to the Education and Training 2010 work programme and particularly the European Framework of Key Competences. Under this action various networks have been set and some events have been organised, contributing to peer learning and the exchange of information. 20 Also, through dissemination and exploitation KA4 of the Transversal programme seeks to support the Lisbon agenda and the implementation of the Education and Training 2010 work programme. 21 Contribution of the sectoral sub-programmes The LLP is also linked to the OMC through its sectoral sub-programmes. They can provide an input to policymaking at the EU level as well as a testing ground during the policy implementation. One example from the interview programme is quality assurance in VET, whose implementation is supported under the LLP. However, the link between the sectoral sub-programmes and the EU policy in the area of education and training is weaker compared with the Transversal programme (especially KA1). Therefore, the link between policy making and the Programme implementation could be improved in the future. For instance, this is relevant in the area of adult education, where the Adult Learning Action Plan could be better linked with the Grundtvig sub-programme. Also, the Programme is contributing to the training and education policy indirectly through many partnerships and networks supported under the Programme (the bottom-up approach). According to the LLP Activity Report, successful school partnerships and other projects and networks under Comenius have 17 Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p Commission staff working document accompanying the communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, An updated strategic framework for European cooperation in education and training. Impact Assessment, Brussels, SEC(2008) Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p EU-level case study on Key Activity 2 Languages. 21 EU-level case study on Key Activity 4 Dissemination and Exploitation. 69

70 contributed to raising awareness of the need for cooperation in the domain of schools, thus paving the way for a communication [...] Agenda for cooperation between the Commission and the Member States on Schools. 22 Also, the action category of multilateral networks supports fora for joint discussion and cooperation in promoting innovation and good practice, contributing to policy making in specific thematic areas (e.g. under the Comenius or Grundtvig sub-programmes). In addition, the Multilateral Networks funded under Erasmus highly contribute to the cooperation initiatives in the area of higher education, especially the Bologna process, the Higher Education Modernisation Agenda, the European Credit Transfer and Accumulation System. As it was expected in the Inception Report, there is some evidence that institutional performance of the beneficiaries supported by the Programme affects European cooperation initiatives at the system level. Several projects aim at harmonising sub-programme activities with general EU policies, e.g. in Leonardo, the ECOTOOL project seeks to harmonise Europass with EQT and ECVET. The review of the National Reports revealed weak links between the sectoral sub-programmes and national education and training policies in some Participating Countries. Therefore, the National Agencies call for more attention to national policy objectives, taking into consideration differences in national education and training systems, as well as closer involvement of national policy makers. Figure shows that the Grundtvig sub-programme was less affected by national lifelong learning policies compared with other sectoral sub-programmes in the period. Figure The extent to which the Programme implementation is affected by national policies Source: summary of assessments provided in Yearly National Agency Reports. Note: 5=high level of complementarity; 4=minor positive influence; 3=no influence; 2=minor negative influence; 1=major negative influence. Better links between the Programme and national lifelong policies are important, taking into account higher political ambition of the EU 2020 strategic framework and national-level commitments (especially quantified targets to be set by the Member States in the area of education and training). Since the Programme and certain domestic policy instruments are implemented in parallel in a number of the Member States, they could be better co-ordinated at the national level in the framework of the EU 2020 strategy, national reform programmes and lifelong learning strategies. The contribution of the sectoral sub-programmes to the EU policy in education and training will be elaborated on the basis of the synthesis of the National Reports prepared by the National Authorities. Contribution of the LLP to the Bologna, Copenhagen processes and development of EHEA Three overarching objectives of the Bologna process have been: introduction of the three cycle system (bachelor/master/doctorate), quality assurance and recognition of qualifications and periods of study. Each objective is addressed by the LLP as a whole or by its separate programme actions, mostly through various established networks. The Erasmus student mobility for studies action contributes greatly to the establishment of system of comparable degrees. For example, the UK supports institutions in using funds under the Organisation of Mobility heading in their Erasmus grant to introduce the Diploma Supplement and through administering the UK Bologna Promoters who advise institutions on issuing the Diploma Supplement. According to the case studies, in the area of credit transfer and accumulation, the LLP student mobility programmes had an important impact on the introduction of the European Credit Transfer and Accumulation System (ECTS). ECTS makes study programmes easy to read and compare for all students, both local and foreign, and 22 Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p

71 therefore facilitates mobility and academic recognition. The system also plays an important role in curriculum design and in validating a range of learning achievements (academic or not). It is also increasingly recognised that qualifications frameworks bring together various elements of the EHEA, including learning outcomes and ECTS credits. National qualifications frameworks (NQFs) have a pivotal role as the nexus where national reforms articulate with the Bologna Process/EHEA. Equally, NQFs provide an articulation between local developments in universities, with national developments and context. In other words, the LLP plays a significant role in narrowing the large difference in the speed of implementation of the Bologna process between individual countries. The international Erasmus networks have an exclusive role to play for catching up countries the primary intention is to bring the Bologna objectives to the institutional level, while the other instruments of the OMC (expert groups, etc. of the Transversal programme) had already played their part for bringing these objectives at the forefront of national policy documents. Although traditionally it is viewed that networks are set up for EU policy implementation, but also they are expected to influence policy making. As a result, the visibility and promotion of networks, especially at the national level, should be improved. Despite existent efforts devoted for the dissemination of the results, better follow-up and long-term monitoring of results is necessary to be established in the future periods of the LLP management. Several drawbacks worthy separate reference: - Promotion of mobility. There is no denial that the LLP significantly promotes mobility. Still, a lack of progress in student mobility area persists within some (mostly Western European) countries. Moreover, it seems that the lack of financial support to Erasmus programme plays a key role in many countries, resulting in an east-to-west imbalance of student mobility within Europe. - Synergy between the European Higher Education Area (EHEA) and the European Research Area (ERA). Since higher education Ministers meeting in Berlin in 2003 and the subsequent strengthening of the link between EHEA and ERA, links between the LLP and the Framework Programme should also be improved. It is very important in the light of the case studies findings, revealing that the Erasmus academic networks often serve as a first step towards developing international research projects for FP. - Dialogue between universities and employers. Although the LLP contributes to University-Business Cooperation via various direct actions, such as the Cooperation between HEIs and enterprises, and also integrates this objective into other actions (e.g. the Erasmus Networks). It was found that the involvement of industry associations in HE networking projects has proved to be difficult and there is obvious need for good dialogue mechanisms and a University-Student-Employer cooperation model for student placements. - European dimension. Mobility from other parts of the world towards the EHEA has increased substantially and faster than international mobility has grown worldwide (EHEA countries attracted 30% of the world s foreign learners in ). Yet internationally mobile learners choose to study in countries and institutions without considering if they are part of the EHEA. Better promotion of the EHEA brand outside EHEA is needed. Further developments in the forthcoming periods. The new EU 2020 strategy puts pressure on new issues that the LLP has to put on the agenda as well such as discussion on the sustainability and greener society; more research-based Europe and stronger role of research based activities. As well the EU added value and issues of EHEA are ever more important hence more pressure on the international activities and EHEA brand (as well outside EHEA borders). The social partners called for closer links between the programme objectives and project outcomes. On the one hand, the priorities set by the Commission are very clear, and the links with policy are quite sufficient. However the situation is different at the level of projects. The feedback loop is interrupted how the projects results are reached and communicated. There is a need to actively exploit the results at the European level to drive the outcomes to the policy making. In other words, two priority issues stressed by the social partners for better linking the policy and programme are: 1) The need for building closer links between the policy and projects outcomes by sharing the content 23 CHEPS / ECOTEC. The first decade of working on the European Higher Education Area The Bologna Process Independent Assessment. Volume 1 Detailed assessment report

72 and results of projects; 2) More transparent way on how the priorities of the Programme and its actions are set up. The stakeholders (students and universities, associations) should be closer involved in the priority setting process. The LLP has contributed to the Copenhagen process through the sectoral sub-programmes, especially Leonardo da Vinci. For instance, the specific calls under the Leonardo da Vinci sub-programme supported projects intended to test how to effectively implement the EQF and how to apply the learning outcomes approach. Such projects address the development of national qualification frameworks, bridging VET and HE, promote validation of informal and non-formal learning and involvement of stakeholders, etc. The multilateral Leonardo da Vinci projects (Development of Innovation and Transfer of Innovation) and networks were used to implement the Copenhagen agenda addressing common tools to facilitate European mobility, European cooperation in VET, and the attractiveness and quality. According to the interviewees, a big bulk of the Leonardo da Vinci budget is spent on setting conditions for transnational cooperation. In the long term, this process, assuming that it is continued, will increase open mindedness and contribute towards slowly changing educational structures. Bilateral and Multilateral School Partnerships Multilateral Projects and Networks and European In-service Training Courses under Comenius contribute to the cooperation initiatives in the area of school education, such as An Agenda for European Cooperation in Schools, key competences, the quality of teachers, etc. The Grundtvig programme actions addressed topics such as putting learners at the centre of the learning process, opening up better transitions between learning pathways, promoting flexible and innovative learning / pedagogy, the acquisition of key competences and particular basic skills, and effective concrete measures. The Grundtvig sub-programme also directly contributed to the Action Plan on Adult Learning by financing many of the activities relating to the development and implementation of the Action Plan, and indirectly through the innovative projects, partnerships, networks and mobility activities funded by the subprogramme. 4. EFFICIENCY Characteristics of the Programme implementation It is important to highlight several characteristics of the implementation process, ensuring the Programme s contribution to the EU policy (the differentiated approach; annual or multi-annual policy priorities; exchanging good practices). A first factor is the differentiated approach to implementation of the Programme. A number of available implementation tools (general/specific calls for proposals, calls for tenders, the framework procedure and other tools) are applied under the Programme to support the implementation of main European cooperation initiatives. According to one interviewee, this approach should be kept in the future: different implementation tools should be selected according to specific needs. Yet one should take into account that the tools of calls for tenders and framework contracts are faster and more responsive to the specific policy needs of the European Commission, whereas calls for proposals can sometimes produce unexpected results (e.g. some priorities may not be addressed by the beneficiaries). Another characteristic of the Programme implementation is annual or multi-annual policy priorities for the calls for proposals. The establishment of these priorities made it possible for the Programme to support main cooperation initiatives launched at the European level. 24 This implementation tool should be kept and coordinated with the three-year work cycles of the Strategic Framework Training and Education Also, since the EU 2020 strategy provides for EU-level and national targets in the area of education 24 Education and Culture DG. Lifelong Learning Programme : Activity Report for 2007 and , p Priority areas for European cooperation in education and training during the cycle of are outlined in Annex II of this document. Council conclusions of 12 May 2009 on a strategic framework for European cooperation in education and training. 72

73 and training, the Programme should be flexible enough to accommodate national priorities of the Member States. It is expected that this could increase national ownership of the Programme. 26 The activities of exchanging good practices based on empirical evidence should continue to be emphasised during the Programme implementation, in particular under the projects and actions directly contributing to the EU policy in certain specific and thematic areas. They allow raising the competences of various stakeholders and facilitate reaching their consensus on various policy issues. However, there is a need to make evidence and learning more specific and thematic in the OMC process. Areas of the Programme implementation to be improved It is important to mention several areas where the Programme implementation could be improved in view of increasing its contribution to the EU policy (dissemination and valorisation; strengthening the involvement of national policy makers and other stakeholders; the action of studies and comparative research; monitoring the Programme impacts on policy). There is a need to improve the visibility of policy cooperation process and results (study visits, peer learning activities, good practices) towards policy makers and other education and training stakeholders through dissemination and valorisation. According to the interview programme, the know-how of dissemination management should be enhanced based on the analysis of existing experience and the exchange of good practice. According to the survey programme, main obstacles to the dissemination, application and implementation of the project results are insufficient budget for using more or other dissemination instruments as well as resistance to change by certain target groups of exploitation. The dissemination of project results could be improved through European networks as well as KA4 of the Transversal programme. Also, the involvement of various stakeholders should be strengthened during the Programme implementation, in particular under the actions of policy cooperation and actions or projects directly contributing to the EU policy. However, this requires additional efforts of all responsible institutions (the European Commission and the Member States, which are responsible for the organisation of peer learning events) as well as the project beneficiaries. Moreover, insufficient links between the sectoral subprogrammes and national policies as well as national-level commitments to the new EU 2020 strategic documents make it necessary to better involve national policy makers in the Programme design and its implementation. The existing mechanisms of the Programme are not sufficient to address this need. As well closer involvement of the social partners (such as student, university associations) is required in the priority setting process. The good practice example would be the unions working the University-Business Cooperation initiative, the results being used in the policy making. More specifically, the implementation of the action of studies and comparative research under KA1 should be improved, in view of increasing its effectiveness and efficiency as well as reducing the implementation gap between the purpose of this action and the quality of project proposals. The alternative instruments of implementation (such as specific calls for proposals, calls for tenders or the framework procedure) or policy (in particular the 7 th Framework Programme) could be considered for achieving the objectives of this action. Also, the activation of studies and research could be integrated in other actions of the Programme. Finally, the Programme s contribution to the EU policy in the area of education and training is not measured by the monitoring system. Hence there is a need to improve data collection and analysis on the basis of the monitoring indicators. Based on the Implementation Measures of Impact Monitoring it is planned to measure the impact of the Programme on policy and practice. 26 Proceedings of the workshop Let s implement together during the Annual conference of the Lifelong Learning Programme Looking beyond May 2010, Barcelona. 73

74 2. Dissemination and Exploitation of the Lifelong Learning Programme Results 1. INTRODUCTION This EU-level case study investigates the practice of dissemination and exploitation (D&E) of Lifelong Learning Programme results. The case-study will contribute to the investigation of the success factors in sharing good practice and making project results as widely known as possible to potential users. The case study will focus on Transversal Key Activity 4 Dissemination and Exploitation of Results and, to a lesser extent, on accompanying measures. 27 In addition, D&E, where appropriate, will be discussed in more general terms. Dissemination and exploitation of LLP results constitutes both a horizontal as well as a vertical LLP objective which ranges from an operational (sub-programme level) to an intermediate / global objective, involving centralised as well as decentralised actions. For the purpose of this EU-level case study, however, we focus on centralised D&E activities only. The overall purpose of this case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the D&E of LLP results. This case study is based on two types of research: - In-depth interviews with the relevant EU-level authorities (Commission / Executive Agency) and representatives of targeted / relevant institutional beneficiaries that have received EU support in the framework of Transversal KA4 and accompanying measures - Desk research, which seeks to analyse actions implementation statistics, EU policy documents, previous studies and evaluations of LLP sub-programmes, LLP activity reports and other relevant documentation. The interviews and desk research were carried out during the period of April 19 - May RELEVANCE The main objective of this section is to describe the European added value of the selected action To what extent do the intermediate and specific objectives of the actions focused on dissemination and exploitation remain pertinent to the EU policy priorities? The objectives of KA4 and other actions aiming at D&E remain unmistakably pertinent to a wide array of EU policy priorities, not only in the LLP framework. D&E cuts across all policy areas and programmes. It enhances congruency between sectors and programmes, sustainability of projects and it contributes to awareness and knowledge of EU programmes and policy. The prime objective of actions aimed at dissemination and exploitation of LLP results is to help create a framework for the effective exploitation of the results of the LLP and previous related programmes at sectoral, regional, national and European levels. Looking at the different priorities of actions, and taking into account the comments of interviewees regarding the background of the actions, the objectives of the different D&E activities can be summarised as follows: 1) to give more results to results; increase follow-up to results after a project s lifecycle is finished; 2) to enhance the feedback process between the project and the policy level (mainstreaming); and 3) to transfer results to other sectors / regions by adaptation (cross-sector fertilisation). Of specific importance for the LLP is Transversal Key Activity 4. The Decision establishing an action programme in the field of lifelong learning 28 specifies key activity 4 (KA4) dissemination and exploitation of results under the Transversal program. It furthermore elaborates on the role of the European Commission in increasing awareness of the LLP to facilitate its implementation. The Decision states that: [t]he Commission, in cooperation with Member States, shall ensure the wide dissemination of information, publicity and follow-up with regard to actions supported under the Lifelong Learning Programme. 27 Accompanying measures support various activities which, though not eligible under the main sub-programmes, will clearly contribute to achieving the Lifelong Learning Programme s objectives. 28 NO 1720/2006/EC. 74

75 The call for proposals LLP puts forward that Transversal KA4 reflects a growing awareness amongst both policy-makers and practitioners of the need to secure maximum impact from EU-funded projects and action in support of the revised Lisbon agenda and delivery of the Education and Training 2010 work programme, thus clearly placing dissemination and exploitation in the wider EU policy framework. The actions funded through this key activity supplements the actions on dissemination and exploitation of specific results within the four sectoral sub-programmes and other key activities and the activities funded under accompanying measures within the four sectoral programmes and KA2 Languages. All interviewees indicated that the objectives of the D&E activities remain pertinent and that it is crucial for the success and quality of the LLP as a whole to prevent programme results from going to waste. There is a general feeling that even more emphasis needs to be placed on D&E. Actions to disseminate and exploit the results of the projects supported by the LLP are of importance in maximising the effectiveness of the projects and their results. Dissemination and exploitation are not enshrined in specific EU policy frameworks, but clearly form a constant theme in all European-level programmes. The LLP aims to have a structural effect on European education systems, by promoting cooperation and innovation across different countries and sectors. This can only be achieved by following up on results of projects (which the accompanying measures and project-owned dissemination plans do) and building a framework within which results and experiences can be exchanged and institutionalised (which is the aim of KA4). All interviewees underscore this aspect of ensuring follow-up to and sustainability of project results as being of key importance in achieving structural and long-term change. This is especially emphasised because experience has shown that in practice this aspect is often not given enough attention by project managers To what extent do the objectives of the actions focused on dissemination and exploitation remain pertinent to the lifelong learning needs of the target groups? The objectives of the KA4, accompanying measures and other D&E-related activities remain very pertinent to the needs of the target beneficiaries and final target groups. Beneficiaries continue to need the support D&E actions provide in order to pay sufficient attention to the sustainability of their projects and final target groups learn about LLP project results thanks to D&E projects. The target beneficiaries of D&E activities are in the first place project beneficiaries from the various subprogrammes. Beneficiaries are required to disseminate and exploit their results within their own projects, they can use an accompanying measure to carry out additional D&E activities and they can use the KA4 to build up more sustainable structures of exchange. The final target groups of dissemination and exploitation vary from project to project, depending on the aims and nature of the specific programme / action. Dissemination and exploitation activities can target potential individual programme participants, institutional participants of the LLP, other educational institutions, commercial organisations and regional, national or European policy-makers. Regarding mainstreaming of results, European organisations and networks seem to play an important role in involving policy makers and other key stakeholders at European, national and local levels. Regarding KA4, the simple fact that many KA4 project applications have to be rejected indicates that there is a clear need for the KA4 action. For instance, in 2009, projects with a mark equal to 75% had to be rejected and 5 additional projects were carried out via the reserve list. In general, support for dissemination and exploitation activities remains crucial for project beneficiaries. Various interviewees point out that individual beneficiaries often find it difficult to focus attention on effective dissemination and exploitation strategies. They are also seen as frequently lacking the expertise and network to effectively make use of their project results. The emphasis that accompanying measures and especially the KA4 place on the cooperation between different organisations is therefore crucial and highly pertinent. By taking part in large-scale D&E activities, beneficiaries can learn from each other and use each others networks to reach other actors, for example on policy level. For the target groups of D&E activities that are not already embedded into the LLP, such as policy makers or other potential final target groups, dissemination and exploitation is the only way they can hear about, and learn from, the results that come out of LLP projects. Especially the KA4 can potentially create structures of inter-european cooperation within the educational field that can ideally increase the quality of education in 29 Lifelong Learning Programme General Call for Proposal

76 several countries. All interviewees agreed on the importance of giving more results to results in a structural way and on the rationale behind KA4 and other dissemination and exploitation activities. While for institutional participants as well as for external target groups, the objectives of KA4 and accompanying measures certainly remain pertinent, it can be questioned whether the means provided match the aspirations. Especially regarding KA4, some interviewees doubted whether the right organisations can be involved and enough energy can be invested in projects as the funding is limited to and projects do not run longer than three years. This point will be revisited later in this case study The European added value. Would other schemes / instruments provide enough support to dissemination and exploitation activities, if there were no actions focusing on dissemination and exploitation? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? Added value can be understood as the value resulting from an EU intervention that is additional to the value that would have resulted from intervention at national or regional level. Information collected for this case study indicates that the European added value is substantial, though there is still plenty of room for improvement. European added value is found in actions that cannot be sufficiently undertaken at Member State level, and therefore, for reasons of scale or effects, are better undertaken at Community level. Dissemination and exploitation supports European added value by ensuring that LLP results and its predecessors are appropriately recognised, demonstrated and implemented on a wide scale. Providing information on the quality, relevance and effectiveness of LLP results seems to almost naturally contribute to European added value. Transversal KA4 shows a clear added value since no other schemes or instruments exist that stimulate crosssectoral D&E on a European level. By its very nature, it seeks to create European networks and communities to diffuse and capitalise on results. Although KA4 only takes up 6% of the total Transversal programme budget and accompanying measures aimed at communication, dissemination and exploitation are fairly limited in number and budget size, 30 the information collected for this case study suggests that the European added value is substantial. By coordinating these actions on a European level, stakeholders from different countries and different levels who are active in the field of education can be involved in the activities. For beneficiaries, the European support for a project also adds additional status to their activities. They report that being able to show EUfunding gives them additional clout also on a national level to mobilise stakeholders and find interested parties, both in the field of education and at policy level. Finally, the mandatory valorisation plan for project proposals has and will likely continue to make stakeholders more aware of the need for D&E and the advantages of involving key European organisations in order to effectively spread and mainstream project results. It is important that these activities are planned and coordinated within the European consortia that also carry out the project themselves. As several interviewees pointed out, project consortia should be free to spend their budget as they see fit which will enhance the feeling of project ownership. Because the funding system is grant-based the project consortia are the complete owners of the projects and their results What evidence does exist of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? Judging from the programmatic framework and the input provided by beneficiaries and officials, it is concluded that there is no overlap between the different D&E activities. Especially KA4 has the potential to encourage synergies between the different sub-programmes, though this potential is not exploited entirely yet. 30 LLP Activity report

77 D&E activities take place at different levels within the LLP: 1) Compulsory valorisation plan for projects funded under the sub-programmes and Transversal programme 2) Accompanying measures for each sub-programme 3) Transversal Key Activity 4 4) Additional Commission/EACEA D&E activities, such as conferences, databases etc. Despite the different levels of action, no obvious examples of duplication of efforts were identified during this case study analysis, as the different actions complement each other rather than duplicate themselves. The different levels are separated clearly from one another and add to the overall framework. The dissemination and exploitation (D&E) plans that every project has to set up encourages project holders to take into account D&E priorities from the very beginning of the project cycle and to reserve resources for D&E activities at the end. The accompanying projects at sub-programme level are connected to specific finished projects, aim to broaden D&E activities undertaken already and are usually limited to one year. The KA4 aims to support large-scale projects which can create a structural addition to the European educational landscape and are not necessarily linked to just one specific project. Finally, the dissemination and exploitation activities undertaken by the Commission and the Agency, such as the project database EVE or the annual Lifelong Learning Conference contribute to the general publicity of the programme and internal exchange of experience. Regarding synergies, it is especially KA4 that can lead to a closer connection of the different subprogrammes. KA4 is, such as all transversal activities, set up as a cross-sectoral activity and should therefore contribute to synergies between different sub-programmes. Most of the projects that have so far been carried out under KA4 follow up on projects carried out in one of the sub-programmes. The assumption that KA4 contributes to synergy can thus easily be made. In 2009, 67% of projects funded under KA4 addressed the specific objective to promote European cooperation in the fields covering two or more sectoral programmes. The interviewed officials confirm this impression. According to the officials, the integrated LLP in general has generated more synergy between the different programmes to which KA4 contributes through networks, internet portals, conferences etc. The preconditions for the coherency of effort at the European level are created by the cross-sector requirements for KA4 projects, the European-wide scope of KA4 key partners (multiplier effect) and efforts by the Agency to put thematic equals into contact with each other to improve networking and enhance the cross-sector element. However, taking into account the limited financial scope of KA4, it can be suggested that the synergy process can be exploited to a greater extent. Most beneficiaries / experts indicated that the organisation of the action is adequate, but that there is limited interplay between the different actions and programmes when it comes to D&E of projects results that potentially have thematic affinity. No mechanism is in place to structurally coordinate such projects across the sub-programmes. As a consequence, the possibility of centralised thematic monitoring across the programmes was suggested by one expert. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the level of implementation of KA4 at the EU level? What is the extent to which the beneficiaries are satisfied with the outputs and results of the KA4? What explains the match between the needs of beneficiaries and their satisfaction with the outputs and results of the actions? Overall, good progress is being made towards the implementation of D&E through KA4. The funds are usually used up to 100% every year. The first round of projects (started in 2007) has now come to an end. Projects generally receive good ratings from the Agency (often 9 out of 10). In 2009, selected applications with a mark equal to 75% had to be rejected and 5 additional projects were carried out via the reserve list. Moreover, none of the projects have ever been terminated prematurely. 77

78 In 2007, about half of the selected applications in the KA4 framework were related to specific objective to encourage the best use of results. 31 A total of 32 D&E issues were integrated into the various subprogrammes, most notably in Grundtvig and Leonardo da Vinci MPs (14 in total). 32 In 2009, specific objective encourage the best use of results was again the most prominent within KA4, both in terms of applications received (65%) and in applications accepted (78%). The majority of applications received and accepted under KA4 related to the key activity s specific objective to promote European cooperation in fields covering two or more sectoral sub-programmes. 33 Several trends in the KA4 framework became apparent in the LLP period 34 : - In 2007, a total of 12 multilateral projects (110 organisations involved) were supported by KA4. 35 (with exception of 1, all projects were funded for the maximum of 75%). Punctual events (conferences) were the most commonly used instruments for dissemination. - In 2008, a total of 12 multilateral projects (113 organizations) received funding under KA4. In comparison to 2007, a broader approach was visible. Nonetheless, the Activity Report pointed out that as far as the involvement of networks and policy makers is concerned, there is still a lot of room for improvement. According to the officials interviewed for this case study, in 2007 and 2008, project promoters still had to be explained the concepts of dissemination and exploitation of results and especially the distinction between dissemination, exploitation, multiplication and mainstreaming. - In 2009, a total of 9 applications were approved (out of 65 applications, 14% success rate) ( 2,373 million). 36. These included: 7 Multilateral projects ( 1,988 million awarded) and 2 Studies and Reference Material ( awarded). 37 Judging from the projects that have been awarded grants, we can thus conclude that the quality of the output of KA4 is improving. The interview respondents confirmed that projects are focusing more on longer term strategies instead of separate events, are involving more large-scale European networks and thus conforming more to the KA4 objective of creating a structural framework within which project results can be disseminated and exploited. Taking into account that the first projects of the KA4 have only just been completed, little information is available on the achieved results. Based on the information collected during the case study, we reach the conclusion that the main factor that directly influences the outputs achieved by KA4 projects is not necessarily the budget size, but rather the knowledge and expertise of the project team and the composition of the consortium. Cross-sectoral KA4 projects require a certain number of partners that have European networks at their disposal and who are knowledgeable about D&E. European associations are increasingly involved in the projects which makes it possible to reach, target and link regional and local levels. In some cases, national and / or local authorities are involved as well. KA4 projects aim to support or build on pre-existing projects and D&E activities within the sub-programmes. The project outcomes vary from very specific ICT tools to building networks and communities for learning. According to some experts, it might be helpful to make a distinction between targeted outputs (products, specialisation) and structural outputs (policy context) and make use of this distinction in the distribution of funds. Thereto, the Commission s Call for Proposals could include a typology of prospects to specify the desired outcome types and (sustainable) impact expectations. While the projects receive good ratings in their own terms, it can be questioned whether an activity like KA4 can have the desired long-term impact. Two factors play a role in this: expertise of project coordinators and budget size. Dissemination and exploitation is a long process that requires specific skills. KA4 is limited in the funding it provides and furthermore does not support projects that run longer than three years. This can have as a result that for some European organisations it may not always be attractive to make use of KA4 31 Specific objective of the LLP, number 11: To encourage the best use of results, innovative products and processes and to exchange good practices in the fields covered by the Lifelong Learning Programme, in order to improve the quality of education and training. 32 DG EAC (2007). Statistics selected applications. 33 DG EAC (2009). Statistical report Selection. 34 DG EAC / EACEA ( ). Statistical reports Activity Report - Synopsis 36 Statistical Report Selection (2009). Lifelong Learning Programme KA4. 37 DG EAC / EACEA ( ). Statistical reports 78

79 funding. For larger organisations, a grant of that has to be distributed within a consortium may not be worth the investment that has to be made for an application. However, for the action to have an impact, precisely these larger organisations need to be involved in the activity, since they have the expertise and network to intervene effectively What are the main (external) factors affecting the outputs and results of the actions? The factors that most prominently affect D&E are not necessary external in scope, but range from internal management at EU level to implementation at project level. The compartmentalisation of the educational sector throughout Europe poses somewhat of a problem. Especially since cross-sector (thematic) monitoring is fairly underdeveloped. On the other hand, cross-sector cooperation, most notably between Grundtvig and Leonardo da Vinci, is taking place and this can positively affect the outcome of D&E activities. Effective D&E of LLP results depends on creating linkages across the sub-programmes and on fishing out and bringing together those D&E projects that could constitute thematic unities. Expertise about dissemination and exploitation is not always equally present amongst beneficiaries, but also at Commission and Agency level. The concepts of dissemination and exploitation, and connected concepts such as mainstreaming and valorisation, are frequently not clearly defined. Sometimes, D&E are confused with communication and publicity activities. It has been suggested that projects need specific training in dissemination and exploitation in order to provide for a common effective approach to these issues. The long-term nature of dissemination and exploitation activities. Structural dissemination and exploitation activities are inherently long term and their impact only becomes visible after a longer period of time. The KA4 budget and project lifecycles are factors that influence the scope and the sustainability of the projects. KA4 allows for a maximum grant of and a project lifecycle of maximum 3 years. While this is, compared with the accompanying measures, already a reasonably high amount of funding, beneficiaries and experts have indicated that structural funding might be necessary to achieve the objectives set out for KA Effectiveness in reaching the target groups and disseminating results To what extent are the target groups of the dissemination and exploitation actions reached? The information collected from statistical data and stakeholder interviews indicates that both with regard to the target beneficiaries as well as regarding the final beneficiaries, the reach of the target groups is satisfactory and improving. In the KA4 framework, the number of submitted applications (54 in 2007, 45 in 2008, 65 in 2009) significantly outweighs the amount of funds. This indicates that the target beneficiaries of the different D&E actions are being reached. In 2009, as previously indicated, selected applications with a mark equal to 75% had to be rejected and 5 were put on the reserve list. Moreover, none of the projects have ever been terminated prematurely. It also seems to be the case that the quality and expertise of project beneficiaries is improving: overall the number of KA4 applications has been growing since 2007, the submitted applications significantly outweigh the available funds and the project consortia increasingly tend to include European organisations and networks (with expertise in the field of D&E) which makes for bigger projects with a wider variety of actions and instruments. The KA4 projects develop several D&E tools and programmes that range from internet portals and websites to self-evaluation tools. The general trend is that the projects increasingly start to focus on involving key stakeholders and policy makers in their D&E initiatives to ensure the sustainability and impact of the project. Whether these target groups are reached can at this stage not be said, as the first round of project has only just been completed. Regarding the Commission s own D&E activities within the LLP, it has to be questioned whether target groups are reached. The project database EVE is generally not seen as a working instrument by most 79

80 beneficiaries. This can be explained by the fact that EVE had to be developed while the programme was already in place. While it is in operation now, this is not known amongst most beneficiaries. The fact that EVE is an essentially untargeted instrument may play a role in this. The annual conference of the LLP organised in conjunction with the National Agency of the country holding the presidency seems to reach its target group after all. It does however remain focused on the groups already involved in the programme To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? Projects focusing on dissemination and exploitation can have concrete results as well. This can be a website, a collection of good practices or a new way of exploiting results. These can be seen as the results of D&E activities. As in any programme, KA4 beneficiaries have to provide dissemination and exploitation plans. However, where projects are solely based on disseminating previous project results, the need for own D&E activities is clearly limited in the absence of specific results. Due to its limited budgetary capacity, KA4 can however be seen as a laboratory for interesting experiments, regarding dissemination and exploitation. The project coordinators, in cooperation with the Executive Agency, are responsible for implementation. The Agency draws up mission reports and provides the projects with support and advice. Left relatively undefined is who is ultimately responsible for disseminating the projects results and for ensuring the sustainability of the projects. The Executive Agency contributes by organising meetings between project coordinators, Operational Groups and information sessions. A post-evaluation of the projects results including an impact assessment would shed light on the effectiveness of these actions, but measuring the specific effects of D&E will be a difficult task Is there any evidence that the dissemination and exploitation actions contribute to integration and harmonisation across the lifelong learning programme? What are the advantages and disadvantages of the integration of previous programmes into the LLP? Though it is too early to judge whether harmonisation and synergy is taking place, it is reasonable to assume that D&E activities in general, and the transversal KA4 in particular, contribute significantly to the integration and harmonisation across the lifelong learning programme. An integrated programme calls for an integrated approach to disseminate and exploit results. As mentioned before, the cross-sector requirement for KA4 projects is likely to contribute to the integration and harmonisation across the LLP. One of the components of the rationale underpinning the transversal programme in general and KA4 in particular is to transfer results to other sectors and regions by means of adaptation (cross-sector fertilisation). Transversal KA4 was established specifically to facilitate and complement the LLP in this way. The interviewees that have been involved in European education programmes for some years, mentioned that the integration of previous programmes in one programme promises to have essential harmonisation effects. For this to happen, continuity needs to be guaranteed, as in the past the memory of a programme was sometimes erased when a new programme was started up. As a result, similar projects were funded by different programmes and similar actions were carried out repeatedly. The integration of the programme and the D&E activities carried out within it have the potential to improve this situation. Also central dissemination tools such as EVE should, when functioning properly, add to the integration and harmonisation of the programme. This is the situation on paper. Some beneficiaries have pointed out that in order for these possibilities to come true, more leadership needs to be shown at EU level by EACEA and the Commission, especially regarding the linking of projects and the use of the project databases. The structural integration has created the preconditions for synergy, but the real integration has to be coordinated actively, according to some interviewees. At this stage, the general impression is that the KA4 contributes to the harmonisation of the programme, however not to its full potential. There are indications that more use could be made of the opportunities created by the integration of the programmes. These issues are discussed in the following paragraph. 80

81 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the EU and the beneficiaries level)? Application and selection phase Within the KA4 framework the selection process is rather straightforward. The Commission is responsible for drafting the Call for Proposals and indicating the priorities. The subsequent selection process is delegated by the Agency to independent experts. The list of selected projects is ranked and the number of projects is linked to the budget cut-off point. All the relevant information and documentation regarding the selection process can be accessed via the Executive Agency s website. However, the Agency indicates that they are sometimes contacted by project coordinators with inquiries to which the answers can be found on the Agency s website. This suggests that information is not as easily accessible as it appears to be. One of the interviewees mentioned that because the KA4 budget is rather modest it should perhaps be used more strategically. At the moment, the objectives and priorities are broad and there are no possibilities to select projects on content criteria. To increase the impact of the projects a call for tender procedure according to thematic field could be introduced. By changing the use of KA4 (same budget, less but bigger projects) through calls for tender, the programme would gain more focus and would likely have a bigger impact on end-users and policy makers. Implementation Within the KA4 framework implementation is the responsibility of the project holder. Throughout the process they are supported by the Executive Agency. The beneficiaries are in general satisfied with the dialogue-based counselling approach of the Executive Agency and indicate that the Executive Agency has become increasingly (pro-)active. However, the implementation and management structure could be improved through: - higher managerial autonomy of the Executive Agency; - greater involvement of the Executive Agency in ensuring projects have the right kind of expertise to achieve the results; and - more extensive consultations with the Executive Agency in the policy making phase with the view of ensuring effective feedback on the lessons learned in the implementation phase (Executive Agency officials have gathered a lot of valuable project-level expertise over the past few years). Regarding the first point, it needs to be pointed out that the responsibility for D&E within the entire framework is not clearly assigned. The basic idea of the framework is that projects themselves implement and coordinate their dissemination and exploitation activities. D&E is seen as a bottom-up process. However, it is to be doubted whether all projects have the expertise to do so. In addition, it is the Executive Agency and the national agencies that have the overview of projects carried out. A more active role of the Executive Agency in linking related projects and exploiting results has therefore been suggested by several interviewees. This is however hampered by the fact that on the one hand, the Executive Agency cannot interfere with the selection mechanisms and on the other hand, policy (in the form of calls of proposals) is shaped by the Commission. This division of labour, while guaranteeing neutrality and openness of the programme, has as a consequence that the ownership of the problem of D&E is not defined clearly and thus left to the community of beneficiaries. Policy-learning and feedback loop, according to one of the officials that was interviewed, is limited because the DG EAC does not sufficiently involve the representatives of the Executive Agency in the policy making phase (for instance, in setting priorities, in designing new types of actions, etc.). Hence, in some instances the Executive Agency officials do not assess the implementation of the priorities and guidance provided by the DG EAC as optimal. The Executive Agency is responsible for supporting individual projects with the dissemination of results (under KA4). The Commission is responsible for analysing the impact of the 81

82 programme and designing policy priorities. The two are rather disconnected and it would be more effective if the Executive Agency were to have a more active role in the design of the policy priorities in general. Regarding the centralised D&E activities of the Commission, a more active approach might also be useful. The project database EVE will be linked to LLPLink so that projects, from the very start of their existence, are taken up in the database and can then, throughout the project s duration, update their information. While this is a positive development increasing the efficiency of the management, it is important that the information provided by EVE is relevant to the target groups that might access it. Beneficiaries questioned the usefulness of the database and asked for more of a targeted, human element in it. Project databases can be a valuable tool for exchange of practices and ideas as long as they are user-friendly, up-to-date and functional. Monitoring and reporting The Executive Agency is responsible for monitoring and evaluation of projects. They undertake monitoring visits to check administrative affairs and / or to give advice. The Executive Agency does not interfere with big project decisions. Monitoring consists of: - visits, - reports, - telephonic monitoring / advice, - inviting beneficiaries to the Agency, - mission reports, - letters of recommendation, - independent expert projects evaluations. Beneficiaries noticed that more (frequent) monitoring is taking place and the review process now shows some teeth. At the same time it was questioned whether the monitoring does not focus too much on administrative issues and too little on the content of projects. The role of the Executive Agency is at the moment perceived as simply overseeing whether funds are used in the correct way. An increased focus on the quality of projects and closer thematic cooperation would be welcomed by most beneficiaries Adequacy of management resources. Do the EU stakeholders and (national) beneficiaries possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? In general, the management resources for the KA4 are adequate. The Executive Agency is conforming to deadlines, and beneficiaries report increased contact time with the Agency compared to the pre-2007 period. Regarding the budget, all available funds are disbursed to projects. In fact, additional projects were carried out via the reserve list (see section 3.1), using up money that was not spent in other sub-parts of the programme. However, the budget available for KA4 projects is not sufficient as it is distributed at the moment. Stakeholders suggest to either increase the overall amount of funds or to support fewer but bigger projects. Beneficiaries have also indicated that the general Transversal budget does not match the scale and scope of the programme objectives. The transversal programme aims to have a structural impact and support the horizontal policies, thus being connected to the entire scope of the educational systems. Taking into account this ambition, budget-wise the transversal programme cannot compare to the sub-programmes. The need for higher levels of funding of the KA4 is illustrated by high numbers of good quality applications and relative low success rates A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into Agency and Commission reports? Judging from the data collected for the case study analysis, it remains unclear how and whether the monitoring information generated by beneficiaries is used for management purposes. 82

83 The general impression is that no lessons are drawn from the monitoring activities. The focus is on administrative affairs and less on content. Monitoring largely focuses on compliance with administrative and financial regulations. The assessment of the results and impacts is largely based on the reports produced by the project coordinators, i.e. it is based on self-reporting. In the future, the main rationale behind the monitoring system could switch from inputs and processes based control towards management by results. By increasing the thematic exchange between Executive Agency, National Agencies and Commission, the monitoring information can then feed into policy-making processes (i.e. setting objectives, call for proposals). Interviewees however indicated that the National Agencies (NAs) are only marginally involved in the management of the centralised actions. They are not always fully aware of the programmes that the Executive Agency implements and can therefore not support the Executive Agency in reaching potential target groups, in monitoring and valorisation of the centralised projects results etc. Since 2009 substantial efforts have been made to ensure closer collaboration between the Executive Agency and NAs. However, despite positive experience of such collaboration, it largely remains limited to the exchange of information. The main reason is that the NAs do not have an obligation to support the Agency and generally lack funds for these types of activities. 83

84 3. Transversal Key Activity 2 Languages 1. INTRODUCTION This EU-level case study focuses on the implementation of the horizontal policy of multilingualism and linguistic diversity. This horizontal policy is embedded in the four sectoral sub-programmes and in the transversal Key Activity 2 Languages. The case study aims to assess the contribution of the LLP to the horizontal policy on the following levels: (1) the Key Activity 2 Languages of the Transversal Programme, (2) the centralised actions of the sectoral sub-programmes, (3) the decentralised actions of the sectoral sub-programmes. The overall purpose is to provide in depth insight into the key evaluation issues of relevance, effectiveness and efficiency of the horizontal policy. Some of the decentralised and centralised actions are assessed in separate case studies. By combining the information collected for the purposes of this case study with the information collected in the other case studies, a clear assessment can be made of the contribution of the LLP to the horizontal policy of multilingualism in general. Since the KA2 is most specifically targeted at the objective of multilingualism, it will be the main focal point of this case study. The sectoral sub-programmes contribute, sometimes directly, at other times indirectly, to the horizontal policy. Where relevant, these contributions are assessed. Finally, the overall framework of the horizontal policy and its implementation is considered. The information presented in this case study is based on the following sources: - In-depth interviews with the Executive Agency, the European Commission, institutional beneficiaries and multilingualism experts. In total eight persons were interviewed for the purposes of this case study; - Desk research, which sought to analyse European legislation, monitoring data, implementation statistics, project reports, thematic studies and other documents. The interviews and desk research were carried out during the period of April 19 th to September 2 nd. 2. RELEVANCE The main objective of this section is to describe the European Added Value of the selected action. 2.1 To what extent do the intermediate and specific objectives of the KA2 and the sectoral subprogrammes remain pertinent to the EU policy priorities, with regard to the horizontal policy of multilingualism? The decision establishing an action programme in the area of lifelong learning specifies the overall objective of the Key Activity 2 to be the promotion of language learning. The LLP as a whole also includes an objective on language learning, namely objective (g) to promote language learning and linguistic diversity. Finally the LLP must further contribute to the horizontal policies of the EU by promoting an awareness of the importance of cultural and linguistic diversity within Europe, as well as of the need to combat racism, prejudice and xenophobia. The priorities of the sectoral sub-programmes are less specifically targeted at language learning. Nonetheless, the Comenius and Leonardo da Vinci programmes share the priority to encourage the learning of foreign languages which clearly fits into the multilingualism policy framework. The Erasmus and Grundtvig programmes do not have specific priorities referring to language learning. As the programmes aim at an increase of inter-european mobility and cooperation in their respective fields of education, language learning can still be seen as an implicit priority of these programmes as well. These objectives remain highly pertinent to the EU policy priorities. Language learning and multilingualism are at the heart of the European Union. Article 165 of the Lisbon Treaty which sets out the European Union s authority in the field of education mentions the teaching and dissemination of the languages of the Member States as the first priority of the EU s activities. In fact, the Lingua programme, which was in place 84

85 since 1989 and focused on language teaching, was the first programme that the European Community set up in the field of education. Promoting multilingualism and linguistic diversity is thus clearly embedded in the policy framework of the European Union. More specifically, KA2 and the horizontal LLP policy of multilingualism ties in with the Multilingualism Strategy of the EU. 38 The strategy sets out the objectives of the Commission in the field of language learning and the reasoning behind it. In the strategy, language learning and linguistic diversity is seen as: - Important for EU internally: encouraging cooperation between the Member States, especially after enlargement; making mobility possible; supporting social cohesion, also with regard to third-country migrants; keeping cultural heritage alive; - Important for EU externally: increasing competitiveness of European economy; enabling business to conquer new markets; contributing to intercultural dialogue; - Important for individuals: for playing an active role in society; increasing employability; increasing mobility. The strategy formulates the objective to raise awareness of the value and opportunities of the EU's linguistic diversity and encourage the removal of barriers to intercultural dialogue. It also reiterates the Barcelona objective of encouraging citizens to be able to speak their mother tongue plus at least two other languages. Especially the KA2 clearly fits into this framework. The actions of the KA2 focus on developing new materials for language learning and fostering more cohesion and exchange in the language community through setting up networks and campaigns. It focuses on improving the access and quality of language learning and pays specific attention to those languages that at the moment are less widely used and taught (LWUTLs). It also aims to encourage language learning amongst citizens who are generally less inclined to learn foreign languages. Finally, since 2009 the calls of proposals have focused more attention on employability and language as a key skill, which again fits well into the general framework, including ET2010 and ET2020. Furthermore, the sectoral sub-programmes also fit into this broader framework. Their emphasis on mobility and international exchange is intrinsically linked to language learning. The relationship between mobility and language learning is reciprocal: language can be seen as a barrier to mobility, yet mobility also leads to more language learning. Although the Erasmus programme is the only sectoral sub-programme which has an entire action specifically dedicated to language learning (the Erasmus Intensive Language Courses (EILC)), the remaining sectoral programmes implicitly focus on the same priorities. 2.2 To what extent do the intermediate and specific objectives of the relevant actions remain pertinent to the needs of the target groups? Potential target beneficiaries of the KA2 can be grouped into two groups. On the one hand there are those interested in developing methodologies and learning materials. This can be universities and other pedagogical experts; however, it can also include commercial actors and businesses, such as publishers. It hereby focuses specifically on groups willing to develop material for the less widely used and taught languages (LWUTLs). On the other hand the action is aimed at institutions interested in building up networks and changing the European framework of language learning. This can include any educational institute, but also teacher s associations, networks of policymakers and similar groups. As KA2 is a transversal action, it focuses on all educational sectors and the projects must target users in more than one sectoral sub-programme area. The final beneficiaries of language activities are of course the learners. Hereby, the action focuses special attention on those groups who are traditionally not involved in language learning, including learners with special learning needs. According to the information attained in this case study, the objectives of the KA2 remain very pertinent to the needs of both the target beneficiaries and the end users of the results. From the interviews with experts and beneficiaries carried out for this case study we can conclude that the objects the KA2 pursues both for development of new teaching materials and for the creation of networks and such are shared by the potential beneficiaries. It is recognised that due to social and economic developments within and outside of Europe the demands on language learning and teaching are continuously evolving. Concrete examples are 38 COM(2008) 566 final: Multilingualism: an asset for Europe and a shared commitment 85

86 the challenges created by immigration, the demands imposed by a globalised economy and the need for increasing mobility within Europe. To meet these demands, the organisations and institutions involved in language education need the support from programmes like the LLP to improve the quality of language education and build up a framework within which they can respond to new languages. For end users, i.e. the language learners, good accessibility of high quality language learning opportunities remains unmistakably important. This applies especially to people with little or no experience of language learning, or reluctant language learners, one of the key target groups of KA2. The gap between different levels of education is especially pronounced in language learning. Increasing language proficiency amongst the lowly educated and equipping educational institutions to deal with this challenge thus deserves unchanged priority. With regard to business interests, the Business Forum for Multilingualism in 2008 emphasised the need for enterprises to be able to operate in different languages. 39 Looking at the Decision establishing the LLP, we can state that the activities carried out under KA2 continue to respond to the framework set up in the beginning. These are reasonably broad, such as to encourage the learning of modern foreign languages but also include specific ideas such as support for the learning and use of sign language and Braille. These objectives stay pertinent to the target groups and are implemented in practice by the KA2 and other programme parts which show a consistent adherence to the framework of the programme. The language activities that take place outside of the KA2 can also be judged to be pertinent to their specific target groups. However, language learning is generally not the primary objective of the actions in the subprogrammes. Looking at the calls for proposals of the different actions of the sectoral sub-programmes, it can be seen that support for language learning is in most places seen as an auxiliary objective or demand which makes mobility, partnerships or other cooperation between different countries possible. Language learning is mainstreamed into for example Comenius Assistantships, Erasmus Student Mobility, Leonardo VETPRO and Grundtvig In-Service Training. The target groups of these language aspects are thus the participants of these actions. The survey amongst participating organisations in all the programmes also shows the relevance of objectives of the horizontal policy. It included a question asking for external factors influencing the results of the specific project. The groups of respondents specifying language barriers as having a strong or some influence on the results of their project were substantial: For Comenius, 41% suggested that language barriers played a role, for Erasmus this percentage was highest at 64%, 55% of Leonardo da Vinci respondents specified language barriers as significant and 47% of the organisational respondents participating in Grundtvig projects did the same. Language skills are thus seen as an essential aspect determining the success of participation in the sectoral sub-programmes. As a consequence, the fact that several actions include the promotion of language learning in their calls for proposals is certainly relevant to the target groups and could possibly be made even more specific and extended. On a higher policy level, it was mentioned by some interviewees that the mother tongue plus two objective, which is not a specific objective of the LLP but which dominates the framework of multilingualism within the EU, can be questioned. Taking into account the enormous discrepancies of language proficiency within different social groups in Europe, one single objective for all target groups may not be effective, since for some target groups this objective is too ambitious, whereas for others it is not ambitious enough. 2.3 European Added Value: Would other (national) schemes / instruments provide enough support to multilingualism related activities if there was no LLP? The European added value of the KA2 can be seen as substantial. This can be seen as a result both of the specific objectives pursued by the action and of the general absence of alternatives for beneficiaries. Regarding the objectives of the KA2, it is clear that the KA2 targets the gaps in national policy provisions. The focus on less widely used and taught languages is by definition additional to already existing provisions. Furthermore, support for regional and minority languages, and especially recent efforts to focus on migrant heritance languages, would be minimal in the absence of LLP funding. The target groups of people not inclined to learn languages and learners with special needs are another aspect of the LLP framework which 39 Business Forum for Multilingualism 2008: Companies work better with languages ; Brussels: European Communities. 86

87 adds value to national efforts. As an example, projects developing methodologies for teaching different sign languages to deaf learners supported by KA2 can be seen as unique and not supported by alternative schemes. Regarding the general availability of alternative funding, the information collected during this case study points to the conclusion that for the activities carried out within KA2, no alternative sources of funding are available. Once again a distinction can be made between the development of materials, methodologies, products and the creation of networks and institutional exchange. With regard to the methodology aspect, the stakeholders interviewed have suggested that the funding of the KA2 creates possibilities to develop teaching and learning materials of the highest quality that would not be available in the absence of the funding. It was pointed out that especially in the niche markets of the LWTULs, it is often not economically viable for enterprises to develop high quality products, as the pressure on the price in the market is too high. Thus, the KA2 clearly adds value to the existing situation by creating a space for the development of high quality methodologies and materials. When looking at the support provided for thematic networks and other activities focusing on exchange of expertise and best practice, it can also be stated that the European added value by the KA2 is substantial. These activities are by definition international and the LLP provides a platform for organisations and networks to cooperate in the field of languages. While the interest in international exchange of information certainly exists, organising it on a national level without LLP funding will create practical and financial difficulties. By asking participating institutions to work together with organisations from other countries and different sectors, the programme actively stimulates cooperation which would otherwise be lost. Finally, beneficiaries have mentioned the fact that participation in the LLP and access to EU funding gives their project additional status which makes the implementation significantly less difficult. Thus by being able to show the support provided by the LLP, it is easier for projects to attract additional participants and also to gain additional funding for the future. Thus, the status provided by participation in the LLP can enable projects to exploit their results and eventually continue without the funding of the KA2. The sectoral programmes do not provide support to specific language projects on a structural basis, but have mainstreamed the language priority into the existing actions. It is thus difficult to say whether, in the field of language learning, they provide sufficient added value. The only action specifically targeted at language training is the Erasmus Intensive Language Courses (EILCs). The EILCs provide clear added value since there is no alternative funding available for this sort of linguistic preparation. The language courses take place in countries where less widely used and taught languages are spoken and thus provide a stimulus for participants of mobility actions going to these countries to engage with the local language and culture. Since these participants usually do not follow academic courses in the host language of these countries, and often also do not speak the local language in their placements, it is likely that without the EILC action participants would not be able to learn the local language, according to the officials interviewed. This is reflected in the positive feedback the Commission gets from national agencies and from organisations organising the language courses. 2.4 Coherence: To what extent are the objectives of the KA2 and of the horizontal policy consistent and mutually supportive? What evidence exists of synergies between the relevant actions and different actions within the sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the relevant actions within the sub-programmes? Were there preconditions for coherence of efforts? Multilingualism is a horizontal policy and thus by definition an area in which to look for synergies between the different sub-programmes and the transversal programme. The KA2 complements the sectoral subprogrammes in its pure emphasis on language learning. As it only feature projects that are transversal in nature, i.e. that target two or more of the educational sectors, there should be no overlap between the KA2 and the sectoral sub-programmes. In order to increase the synergy between the different parts of the LLP in the area of language learning, it is rather in the sectoral sub-programmes that changes should be made to increase the emphasis on language learning than in the KA2. The objectives of the KA2 are consistent, however not necessarily mutually supportive. There have only been a few changes in the calls for proposals throughout the years which suggests that no specific problems were signalled in the implementation. 87

88 The KA2 aims to improve the quality and accessibility of language learning in Europe. The specific objectives of the actions and the calls for proposals support this aim. As can also be seen from the other parts of this case study, the KA2 is clearly divided into one part (multilateral projects) focusing on the development of specific products and the organisation of awareness raising activities and another part (thematic networks) focusing on exchange of knowledge and good practice. The interviews with beneficiaries posed the question whether this division works well in practice, as multilateral projects were seen as pursuing entirely different objectives than networks. Looking at the projects that have so far been funded, however, it can be seen that these two priorities are complementary to one another indeed. Thus, networks and accompanying measures can be used to focus on the dissemination and exchange of good practices that come out of the multilateral projects, or, as the projects of the KA2 are not finished yet, earlier Lingua actions. The objectives of the KA2 actions can thus be seen as both consistent and complementary. With regard to the synergy between KA2 and other actions within the sectoral sub-programmes, it can be said that by definition the KA2 adds to the harmonisation across the programme due to its transversal character. Different from for example KA4 (dissemination and exploitation) however, the subject matter of the KA2, i.e. language teaching, does not by definition require a transversal approach. The relationship between the KA2 and the sub-programmes can therefore be seen as one of complementarity the activity provides a space for projects that do not fit into the sub-programmes due to a cross-sectoral approach. As a consequence there is also no overlap between the different sub-programmes and KA2, as they all fit into their specific sectoral category. Nonetheless, the specific focus on multilingualism of the KA2 does give it a special status when compared to the sub-programmes where multilingualism is one of several objectives. There are two aspects that could be improved in the context of the synergy between sectoral subprogrammes and the KA2 in the field of language learning. Firstly, some interviewees questioned whether the concept of transversality, as it is currently defined, reflects the reality of the complementarity between the KA2 and the sectoral sub-programmes. Transversality at the moment refers to projects targeting at least two of the four educational sub-sectors. Though trying to go beyond the sectoral approach, this definition still refers to and thus reinforces the division into the specific sub-sectors. Projects that fall in the grey zone between different sectors are made to fit this sectoral logic which can lead to artificial project designs. More freedom for unconventional approaches that do not take into account different sectors may be advisable. By defining transversality as referring to projects that do not fit into the sectoral framework in general, the complementarity of efforts can be strengthened. Secondly, according to the interviewees, the centralised actions of the KA2 contribute significantly more to the horizontal policy of multilingualism than the sectoral sub-programmes. It provides opportunities for innovative projects that can lead to important improvements in the policy area. This lies in the nature of the programme. Whereas the KA2 focuses specifically on the policy area of multilingualism and language learning, the sectoral sub-programmes have mainstreamed language learning into their approach. Thus, projects and actions under the sectoral programmes that do not specifically target language learning are still seen as contributing to the horizontal policy, such as mobility actions which have a clear impact on the language skills of participants. However, these actions show little complementarity with the KA2. The centralised actions of the sectoral sub-programmes on the other hand could potentially lead to strong synergies between the sub-programmes and the KA2. However, these actions, such as multilateral projects and networks, only sporadically pay specific attention to language learning. While it is unquestionable that the sectoral sub-programmes contribute to the horizontal policy of multilingualism, the synergy between the sectoral programmes and the KA2 could be increased by strengthening the language dimension throughout the entire programme and specifically in the centralised actions of the sectoral sub-programmes. 3. EFFECTIVENESS What is the level of the programme implementation at the EU level? What is the extent to which the beneficiaries are satisfied with the outputs and results of the KA2 and other relevant actions? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results of the KA2 and other relevant actions? The implementation of the KA2 can be judged to be effective, as - All funds are used. - The quality of applications is high according to the officials interviewed. 88

89 - The quality of projects is high, though no information about results is available yet. The KA2 receives a high number of applications and is using all the funds available. It supports between 25 and 30 projects per year, awarding between 9 and 10 million of grants per year. The greatest part of the funding goes to multilateral projects (20 in 2009), though the number of networks being supported is rising (7 in 2009). The activity report of the DG EAC concludes that the KA2 succeeded in complementing the work of the sectoral programmes on the promotion of language learning and multilingualism in Europe 40. The quality of applications is generally judged to be satisfactory. Sometimes applications have to be rejected despite receiving high marks, due to a shortage of funds. However, according to EU level officials most applications that are rejected do not fit the strict requirements of the KA2 and are thus not rejected because of a shortage of funding. According to the interviewees, the EACEA for example receives applications that are not cross-sectoral in their set-up and thus should be directed at a sub-programme, or applications that do not have access to a sufficiently varied consortium of project partners. Projects that do receive funding certainly fulfil all these conditions. They fit into the KA2 framework by setting up cross-sectoral initiatives, focusing on the target languages and target groups. At the time of research, the first round of finished KA2 project was still being evaluated. However, some final reports of KA2 were already made available on the website of the EACEA. Looking at both the final reports and progress reports of projects, it appears that most projects are achieving the results that they envisage. Regarding the outputs that projects aim to achieve, it can be stated that a very broad range of results can be expected. Nonetheless, some trends are discernible: in general, a lot of multilateral projects aim to develop new ICT-based methodologies. Unsurprisingly, the Statistical Report of 2009 stated that 24% of the KA2 actions show complementarity with KA3 ICT. Increasingly, the projects focus on non-european languages such as Arabic and Chinese. Especially networks have started involving more business actors and policy makers. While information about concrete outputs and results of projects is not available yet, the beneficiaries and officials at Commission and EACEA stated to be confident about the quality of the projects. Regarding methodologies and materials, it was said repeatedly that the outputs of KA2 actions are much higher in quality than what is already available. It was also said that in comparison to language projects supported by the sub-programmes, the multilateral projects of the KA2 are superior, due to a more specialist focus on high quality language learning methodologies. On the other hand it was also mentioned that especially the thematic networks can be said to contribute significantly to the promotion of language learning and linguistic diversity. However, by setting more focused priorities, even more use could be made of them. The results of some of the multilateral projects on the other hand are frequently only of average quality. According to the experts interviewed at the EACEA and at participating institutions, the success and failure of KA2 projects is mainly dependent on the different project partners and specifically on the management skills of the leading beneficiary. Especially thematic networks, but also multilateral projects, rely on a good consortium of participating organisations in order to aggregate expertise and share the burden of carrying out the project. If the work of a project is only carried out by one organisation, this is often an indication that the project is struggling. Furthermore, inexperienced project managers sometimes cannot cope with the high demands of managing a European project. Lastly, some projects produce high quality results but fail to exploit them or add onto their success. Regarding the implementation of the horizontal policy of multilingualism throughout the rest of the programme, we can first look at the centralised actions of the sectoral sub-programmes. As mentioned before, the sectoral sub-programmes do not provide specific support for projects targeting language learning but rather include this as one of the possible themes and topics of Multilateral Projects, Networks or other centralised actions. According to the survey results, only a rather small fraction (11%) of organisations participating in centralised projects specified linguistic preparation as one of the main activity or output of their project. Nonetheless, linguistic preparation is a very specific activity which excludes many other potential language projects. 40 DG EAC: LLP , Lifelong Learning Programme , Activity report for 2007 and

90 When looking at the project compendia from , it can be concluded that projects directly targeting language learning and linguistic diversity are supported sporadically, and mainly in the sub-category of multilateral projects. More frequently, projects include a language component as part of their overall approach, for example by providing the results in a number of different languages. Regarding the output of these projects, no concrete evidence is available. According to some interviewees however, the quality of language projects in the sectoral sub-programmes lags behind those in the Key Activity 2. In order to achieve the same high standard as the KA2, more targeted selection practices for the language priorities are advised. While projects themselves may not necessarily be targeted thematically at language learning, they still often have an impact on the participants foreign language skills. This can be seen in Table which reflects the responses given by institutional beneficiaries of both centralised and decentralised actions. Table Influence of participation in LLP on foreign language skills of participating organisation Programme Centralised / Decentralised Comenius Erasmus Leonardo Vinci Grundtvig da Centralised Decentralised Centralised Decentralised Centralised Decentralised Centralised Decentralised Strong influence Some influence No influence Do not know/ Cannot answer Staff/trainees 25% 41.7% 25% 8.3% Learners 29,2% 29.2% 33.3% 8.3% Staff/trainees 47,6% 38.5% 8.6% 5.3% Learners 53,3% 35.4% 7.5% 3.9% Staff/trainees 25% 41.7% 25% 8.3% Learners 29,2% 29.2% 33. 3% 8.3% Staff/trainees 39,4% 43.5% 10.6% 6.5% Learners 56,9% 34.1% 5.1% 3.9% Staff/trainees 15% 47.5% 12.5% 25% Learners 28,2% 41% 20.5% 10.3% Staff/trainees 36,2% 41.7% 14.1% 8% Learners 38% 33.8% 16.9% 11.3% Staff/trainees 21,8% 41.8% 20% 16.4% Learners 7,8% 25.5% 37.3% 29.4% Staff/trainees 30,1% 40.8% 20.3% 8.9% Learners 26.7% 34.9% 24.5% 13.9% Here we can see that, regardless of the objectives of the projects of the sub-programmes, participation has, in a majority of cases, at least some influence on the foreign language skills of the participants. The decentralised actions of the sectoral programmes focus on mobility, partnerships and exchanges. The Erasmus Intensive Language Courses is the only action that directly targets language learning. Annually around participants take part in such a course. This action meets the needs of the target groups: according to the survey of the LLP beneficiaries, the language barrier is the most important under Erasmus (12,7% of the Erasmus respondents argue that the barrier of language had a strong influence compared to the average of 7,9% for the whole programme). A lot of decentralised actions, especially mobility actions, target language learning more implicitly. By taking part in one of the decentralised actions, participants automatically improve their language skills. This can be seen in the results of the survey of individual participants, summarised in Table Table Influence of participation in LLP on foreign language skills of participating individuals Programme Strong influence Some influence No influence Do not know / Cannot answer Comenius 57.6% 36% 5.3% 1.2% Erasmus 70.7% 25.2% 3.6% 0.5% Leonardo da Vinci 50.9% 41.8% 6.5% 0.8% Grundtvig 42.8% 41.7% 14.5% 0.9% 90

91 The effect of participation on the beneficiaries foreign language skills can thus be assumed to be substantial (see also the results of the survey of beneficiaries). In the case of Erasmus, this is most prominent, with more than two thirds of respondents reporting a strong influence. Regarding the main outcomes/activities of projects, significantly more respondents name linguistic preparation as one of the outcomes of their projects than is the case for the centralised actions. For Comenius, 47,9% choose this option, for Erasmus 30%, in the case of Leonardo 40,4% and for Grundtvig 22,7%. This shows the important role language learning plays in the decentralised actions. The survey results also, however, confirm the impression that English continues to be the dominant language amongst beneficiaries and that this dominance is rather enforced that diminished. This conclusion is drawn by both experts interviewed in the framework of this case study as well as by the other case studies. Between 40 and 47% of individual beneficiaries of the sectoral sub-programmes responded in the survey that their English language skills had improved as a result of participation in the programme. Other languages that are selected frequently are Spanish, French and German. These are however selected by only around 10% or less of the respondents. Other, less widely used languages, receive very low scores. Regarding the results achieved in terms of the horizontal policy of multilingualism and linguistic diversity, we can therefore conclude the following: - The KA2 attracts very high quality projects, both Multilateral Projects and Networks. Results and outputs are not formally evaluated yet, but are expected to be of high quality. - The centralised actions regularly, but infrequently fund projects that are specifically targeted at language learning and linguistic diversity. The quality of projects is high, but lower than in the KA2. - Participation in projects, both centralised and decentralised, has an automatic impact on the participants foreign language skills. Institutional beneficiaries report that participation in the programme increases the foreign language skills of teachers, trainees and learners in the organisations. Individual beneficiaries of decentralised actions overwhelmingly state that their participation has an influence on their language skills. - The influence participation has on participants language skills, as seen in the survey results, is strongest for English, much less strong for Spanish, French and German and very small for the less widely used languages What are the main external factors affecting the outputs and results of the KA2 and other relevant actions? The discussions with experts, beneficiaries and officials identified several external factors that affect the outputs and results of the LLP in the area of multilingualism. While sometimes hindering the success of projects, these factors are at the same time issues that projects aim to address. There is a general agreement amongst experts that to have a long-lasting impact on the status of multilingualism in Europe, structural changes in national structures are needed. Language learning needs to be even more embedded in education systems, by enabling learners and teachers to improve their language proficiency without losing out on their other diverse professional interests. When talking about motivation for language learning for example, at any level it is seen as crucial that systemic opportunities and incentives are created. These can be simple things like allowing university students of all disciplines to build up study credits in language courses; developments that are already in place in several countries and sectors, but where a lot of potential is still not exploited. However, KA2 projects can only influence these changes to a limited extent as they focus on the European dimension of language learning. The legal base is set up in a way so that Member States stay in charge of their education systems. As a consequence, the effects of European-level activities will always be limited without the active contribution of Member States. The target groups in terms of end users of the KA2 projects are difficult to reach. This is of course by definition the case, as the KA2 aims to support projects that target those learners that are traditionally not inclined to participate in language learning. However, as projects developing methodologies have to involve the end-users, i.e. these specific target groups, in the development of their product, they are dependent on the participation of groups that are defined as hard to reach. This can lead to problems in the implementation of projects. 91

92 The wide-spread dominance of the English language can make it hard for projects to generate sufficient support and interest for their projects. Especially when it comes to policy makers and the business community, the impression that English is the only (foreign) language needed is wide-spread. When trying to find ways to promote the teaching and learning of the LWUTLs, this resistance amongst key stakeholders can predictably affect the outputs of a project. In the context of mobility actions, English is also the dominant language which can limit the impact of participation on participants other foreign language skills. Finally some interviewees have pointed out that, perhaps contrary to what one might expect, in some areas the interest in multilingualism and language learning has actually declined as a result of globalisation and internationalisation. This is seen as an anti-reaction, as a perceived threat to (national or regional) identities can lead to a reaction of what is perceived as foreign influence. Beneficiaries reported concrete examples of problems in generating interest for their activities as a result of this kind of resistance, especially amongst specific regional policy-makers To what extent does the Programme implementation contribute to the EU education and training political priorities (e.g. Education and Training 2010 Programme, the Lisbon strategy)? To what extent does it contribute to the OMC mechanism of European benchmarking and evidence-based policy making, mechanism of peer learning activities and other fora of learning? The KA2 and other centralised actions that fund projects in the field of language learning contribute mainly indirectly and at times directly to the EU educational and training political priorities, through projects that improve the quality of language learning in Europe, work on implementing European initiatives and focus increasingly on labour market issues. Decentralised actions, for example mobility actions, which lead to language learning of participants, contribute directly to the Multilingualism Strategy of the EU. The fact that the objectives and priorities of the KA2 and the horizontal policy of language learning in general fit into the European policy framework has already been made clear. When looking at the projects that the KA2 has until now supported, we see that most of them support the development of language learning methodologies and materials. The general objective of these activities is to improve the quality of language learning and thus finally make more people learn languages and increasing their mobility. These objectives clearly relate to the ET 2010 programme and the European framework of Key Competences. However, as these projects only provide the conditions for people to learn languages, it cannot be said that they contribute directly to the policy framework. However, there are other KA2 projects that focus specifically on the implementation of existing European initiatives. Thus in 2007, a project focused on stimulating the use of the European Language Portfolio by teachers and learners. In 2008, one project aimed at promoting the European Framework of Key Competences for Lifelong Learning amongst parents and children. Another project aimed at enhancing the use of the European Language Portfolio (ELP) amongst different target groups. A third project aims to adapt the ELP for deaf learners and integrate it in the Europass Language Passport framework. 41 In this way the projects contribute directly to the European education framework. Furthermore, in 2008 and 2009 it can be seen that there are more KA2 projects focusing on business concerns and employability regarding languages. Several projects focus on the tourism industry, but other projects formulate general objectives relevant to enterprise development, such as one project which aims to develop an online assessment tool for intercultural communication. These kinds of project specifically aim to increase the applicability of language learning to economic growth, responding to both the ET2010 programme and the Lisbon strategy. Finally, various networks are set up under KA2 and some conferences have been organised which contribute international peer learning and exchange of information. Increasingly, policy makers have been targeted by these activities and beneficiaries have reported success in creating platforms for exchange of ideas and dissemination of good practice. While this does not directly take place in the framework of the existing education OMC, it adds to the structures already in place by creating a new layer of exchange. Decentralised mobility actions and of course the EILC action have direct influence on participants language proficiency. They clearly promote to the EU s multilingualism strategy by strengthening language learning in 41 See Project Compendia 2007, 2008,

93 all sectors and they contribute to the Mother tongue + 2 objective. However, as has been pointed out, the dominance of the English language, also and especially in mobility actions, can be seen as limiting these actions contribution to the preservation of language diversity. Any additional actions, such as the EILCs, do focus specifically on the LWUTLs, which fits into the general multilingualism strategy of the EU How does the implementation of KA2 and other relevant actions contribute to the horizontal priority of multilingualism? The KA2 clearly contributes to the horizontal policy of multilingualism to a great extent. It is purely focused on language learning and linguistic diversity and is embedded in the European framework regarding multilingualism. It is also seen as providing higher quality than other actions focusing on languages. The KA2 projects by definition have to focus on improving the accessibility and quality of language learning in Europe. To give an impression: 24 of the 29 projects supported in 2009 were pursuing the objective to promote language learning and linguistic diversity. Equally, 28 of the 29 projects were targeting the Lisbon Key Competence of Communication in the foreign languages. KA2 is also financially the largest single action devoted to languages, giving out grants of around 9 million per year. As a comparison: in 2007, all the other sub-programmes taken together supported 18 language projects with a total budget of 5,6 million and the national agencies of all the Participating Countries together spent 35 million on decentralised actions linked to languages. 42 The KA2 thus represents an essential part of the entire multilingualism framework of the KA2. As the KA2 is entirely focused on languages, it is also perceived as creating the highest quality projects in this field. This is connected to specific structural characteristics, especially of the application process. Thus the calls for proposals for KA2 projects include specific requirements on the language proficiency of the project beneficiaries. These stipulate that all KA2 project consortia for multilateral projects have to include at least one organisation from a country in which the targeted language is spoken. Also, the expert evaluators making judgments on the quality of applications all have to be experts in the field of multilingualism. Finally, the EACEA staff supporting projects also has a lot of expertise in the field of language educations so that they can give expert advice to projects. In the sub-programmes, these three conditions are not present which, according to the interviewees, leads to a lower quality of projects. Apart from KA2, it can be concluded that mainly the decentralised mobility actions and the specific action of the EILC contribute to the horizontal policy of multilingualism. The centralised actions of the sectoral subprogrammes sporadically fund projects that target multilingualism as an issue. This does however not happen on a structural basis. The national reports show that in most countries the horizontal policy of multilingualism is seen as being automatically embedded in the nature of the programme. As the programme stimulates international exchange and mobility, language learning will always be a central aspect. Regarding linguistic diversity, at several points in the decentralised actions attention is paid to promoting the LWUTLs. Thus, the EILCs do not apply to the widely used languages, and in the Comenius programme applicants applying for mobility and assistantship actions in countries speaking LWUTLs receive additional points for their applications How does the implementation of the KA2 and other relevant actions contribute to the horizontal priority of equal opportunities? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? The KA2 does indirectly contribute to the horizontal priority of equal opportunity. However, there are no mechanisms in place that ensure that projects take account of issues related to equal opportunities. For the purposes of this case study, no information is available for the sectoral sub-programmes on this question. Nonetheless, the activity does target disadvantaged groups in society. Thus, the target group of people traditionally less inclined to learn languages can be interpreted as coming from less privileged backgrounds, migrants a target group of some projects, and several projects focus on learners with special needs, for example deaf people. 3.3 Is there any evidence that the Transversal KA2 and other relevant actions contribute to integration 42 DG EAC: LLP , Lifelong Learning Programme , Activity report for 2007 and

94 and harmonization across the Lifelong Learning Programme? What are the advantages and disadvantages of the integration of previous programmes into the LLP? KA2 is a transversal activity and thus by definition adds to the harmonization of the programme, as actors from all sectors can benefit from it. The integration of the programme has created clear possibilities for more harmonization of the language activities, but these can still be exploited more extensively. The integration of the programme has created important opportunities for synergy in the field of multilingualism. Multilingualism is embedded in all the sub-programmes as a horizontal policy and in KA2 specifically as an objective. Even the decentralised programmes have to pay attention to language activities, especially in the context of mobility actions. There is however disagreement about whether these opportunities are exploited to a satisfactory degree. This has to do with the organisational structure of the programme. As elaborated under section 4.1, respondents noted that more coordination between the sectoral programmes and the KA2 could be achieved through still intensifying the already existing cooperation between the units in Commission and Executive Agency dealing with the issue of multilingualism on the one hand, and the sectoral sub-programmes on the other hand. Increased exchange between national agencies, European Commission and the Executive Agency about the horizontal policy, e.g. on centralised language-related projects and results achieved, could lead to multilingualism attaining a higher status throughout the entire programme. All projects within KA2 have to focus on objectives that are relevant to more than one sector. As a consequence, the beneficiary organisations also often come from different sectors and cooperation with each other in consortia. Beneficiaries of the KA2 were before 2007 frequently involved in projects under the Lingua programme. Additionally, some of the beneficiaries are also involved in projects under the subprogrammes. By connecting beneficiaries from all the sectors and all the different sub-programmes, the KA2 thus certainly adds to the harmonization of the programme and importantly enables beneficiaries from different sectors to learn from each other. However, as has already been mentioned, the subject matter of KA2 is not necessarily cross-sectoral, in the same way that for example KA4 Dissemination and Exploitation is. It is thus not focused on bringing about harmonization in the programme, but it is likely that this is an additional effect of the activity. Most fundamentally, the multilingualism unit of the European Commission determines what is being done in the field of multilingualism. It also writes the calls for proposals for the KA2 and sets the objectives for the activity. However, in this case the transversal nature of KA2 is a limitation for the policy makers at the Commission, as specific objectives targeted at one educational sector, for example early language learning, cannot be embedded within the KA2 framework, but should be integrated in the respective sectoral subprogramme. The objectives for the sub-programmes however are set by the respective units of the subprogrammes and are not focused specifically on languages. As a consequence we have a situation in which the KA2 sets specific priorities for language learning attuned to the EU multilingualism policy, whereas the sub-programmes treat language learning as one of several general issues. This situation requires close cooperation between the different units in the EACEA and the Commission in order to exploit the potential for synergy between the programmes and also open up more possibilities for the policy units to give direction to the language dimension of the programme How successful are the KA2 and other relevant actions in reaching the target groups and institutional beneficiaries? In general, the KA2 is successful in reaching its institutional beneficiaries. Whether the final target groups are reached is difficult to say, as no final project reports have been published yet. In fact, taking into account the nature of the projects, we can say that final beneficiaries will only be reached once the project results, for example a new methodology or product, are used in the classroom. Though the KA2 targets the development of new methods and products for example for reluctant language learners, it does not support projects that directly organise activities for these target groups. As said before, the KA2 receives a high number of applications every year and according to the officials interviewed, the institutions applying for grants are the ones that the activity is targeted at. On the other hand, questions could be raised about the representation of different target groups in the group of organisations involved, as universities make up by far the largest group. To illustrate, in 2009, of the organisations involved in the consortia of supported projects: 94

95 - 41% were universities, - 6% were adult education providers, - 4% were vocational training centres organisations, - 4% were small and medium enterprises, - 4% were non-profit organisations, - 3% were non-governmental organisations. 43 It is unclear whether this dominance of universities constitutes a problem and the interviewees do not come to a general consensus on this question. Two issues are important in this context. On the one hand, it has to be stated that the fact that universities are involved does not mean that all project focus on higher education. Clearly, university actors can also develop projects for other sectors. This is simply a question of expertise and a lot of expertise is located at universities. Also, for the development of learning materials the involvement of end-users is already a requirement for projects. So there is no reason why a consortium dominated by universities should not develop a high quality project for other sectors as well. On the other hand however, the experts and beneficiaries interviewed indicated that other, non-university organisations may feel pushed out of the programme, especially non-educational organisations, such as non-profit organisations or SMEs. As these organisations often bring essentials skills to projects that are not present in for example universities, it is worth considering whether targeted actions need to be carried out to increase the diversity of the group of KA2 beneficiaries. Regarding the sectoral sub-programmes, in the context of this case study, no conclusions regarding the reach of institutional beneficiaries and target groups can be drawn. As said before, the centralised actions attract sporadically projects with a focus on languages or with a language component. However, this is not the result of conscious efforts. If a more conscious effort was made to encourage language projects, possibly more organisations dealing with issues of multilingualism would be attracted to the programme. The EILCs is very successful in reaching its target groups. In fact, the demand for these courses outstretches the supply. Thus, only about 7% of students going on Erasmus exchanges are able to take part in an intensive language course in the framework of this action. The survey data gives some indication on the specific target group of linguistic minorities. Table shows the share of respondents that answered positively and negatively to the question of whether their project was successful in reaching linguistic. Though the majority of respondents did not provide an answer to this question or chose the option not applicable, we can see that respondents from the Comenius and Grundtvig programmes are more positive about reaching this specific target groups than the respondents from the Erasmus and Leonardo Programmes. In general however it seems that those projects that do target linguistic minorities, are successful in reaching this target group. Table Proportion of projects successful in reaching target group of linguistic minorities Very successful Rather successful Rather unsuccessful Very unsuccessful Do not know / cannot answer/ is not applicable All 5,3% 8,8% 2,8% 1,5% 45,8% Comenius decentralised 6,2% 12,2% 2,9% 1,6% 45,8% Erasmus decentralised 2,3% 3,9% 1,7% 0,7% 54,1% Leonardo decentralised 3,7% 4,7% 2,1% 1,1% 43,4% Grundtvig decentralised 7,6% 10,4% 3,5% 1,6% 43,1% All centralised 4,3% 7,4% 1,6% 2,3% 45,1% Since 2007, Erasmus students have the possibility to participate in a placement in another country. Their language preparation is important and might go beyond the basic level needed for social integration. However, the majority of students going on placements cannot benefit from the EILC as these courses are linked to the academic year and placements can start at any time in the year. This important target group is therefore not reached by the courses. Finally, EILCs are also open to participants from other sectoral mobility actions. This opportunity is rarely made use of, since the courses fill up too quickly. The participation of participants from other sub-programmes could thus be encouraged. 43 LLP Statistical Report 2009, KA2. 95

96 3.4.2 To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? The dissemination and exploitation of project results varies to a high extent, but is also difficult to assess at this stage. Though the formal requirements for dissemination and exploitation are met, the question whether these will have effect in practice cannot be answered. While projects have to present dissemination and exploitation plans in order to gain access to grants and it can be assumed that this basic condition is met, according to some beneficiaries, the exploitation of developed methodologies or materials developed for the KA2 does not always have the highest priority. As is often the case with dissemination and exploitation, beneficiaries indicate that it can sometimes be seen as more of a box-ticking exercise than as an integral part of the projects. However, the thematic networks supported by KA2 often focus exclusively on dissemination and exploitation, as they aim to set up structures within which good practices can be shared and experiences can be exchanged. Nonetheless, these multilateral networks could be better linked to the products developed by the multilateral projects. In the context of this case study, no information is available for the sectoral sub-programmes. 4. EFFICIENCY The information collected for the purposes of this case study suggests that for the general functioning of the programme with regard to the horizontal policy of multilingualism, the existing structure is satisfactory. However, there are a number of weaknesses which can be addressed to make the structures function more efficiently, especially with regard to the application stage Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient? The information below refers to the programme management system of the KA2. The framework of the other relevant actions, both centralised and decentralised, are evaluated in other case studies and not directly relevant for the horizontal policy of multilingualism and linguistic diversity. The application stage. The application and selection phase of the KA2 is carried out in the way that is determined by the legal framework. This means that expert evaluators are selected from the database of applicants on the basis of their CVs who are then employed to determine the quality of the projects on the basis of the criteria. The projects are ranked and the best are selected. By following this process, the neutrality and openness of the application procedure is guaranteed. According to the interviewees, the application process is very cumbersome and expensive, but in the end professional and transparent. The criteria which are used to judge proposals are clearly publicised beforehand. Projects get extensive feedback on their application and can re-enter improved versions a year later. According to information provided by the EACEA, about 10% of KA2 applications submitted per year are re-submissions which suggests that the feedback procedure is working well to improve project proposals. While the system involves a lot of checks to safeguard the neutrality of the process, in the interviews with experts, beneficiaries and officials, several weak spots were identified. These focus mainly on the role of the evaluators, and to a lesser extent on the demands placed on applicants. Regarding the evaluators, respondents expressed their doubts about the quality of some aspects of the evaluation system. Firstly, according to the officials interviewed in the context of this case study, the fact that the evaluators are not interviewed or tested before they are taken on as experts, raises concerns about the question whether it can be guaranteed that they are able to perform their task. While the system in place now is supposed to guarantee the objectivity of the selection process, it may have adverse consequences for the quality of experts. However, the system is used throughout the LLP, so unless this issue is raised throughout the programme, this finding should be treated with care. Secondly, the evaluators are perceived by some beneficiaries as being purely academically trained and thus not necessarily able to pass a 96

97 clear judgement on the quality of a project. In fact however, experts do have to have practical experience in the field. The final point about the evaluators relates not to KA2, but to the language actions under the sectoral subprogrammes. Whereas in KA2, the expert evaluators have to have a background in the field of languages, in the sectoral sub-programmes, they are only required to have a background in the respective educational sub-sector. However, the sub-programmes also support language activities. These applications for language activities in the sub-programmes are therefore not evaluated by language experts. According to several respondents, this has led to a situation where the sub-programmes are supporting projects which under KA2 would not get past the evaluator s judgement. This is not just an issue of consistency, as it is easier to get a grant for a language project under the sub-programmes, but also one of efficiency, as grants are given to projects of lower quality. Finally, KA2 beneficiaries have commented on the electronic application form used in the process. The beneficiaries interviewed are under the impression that this application form does not always give projects the opportunity to really explain their approach. The formalisation of the process and the limitation of information that can be provided are perceived by some as focusing on an applicant s ability to write a project application instead of focusing on the quality of projects. The application process is crucial for the functioning of the programme, as it determines the quality of projects and also has a strong influence on the way that beneficiaries perceive the functioning of the programme. Especially the question of well-qualified evaluators needs to be addressed in order to keep the selection process transparent and objective. The implementation stage. The implementation structure of the KA2 is generally working well. No specific problems were identified by the research for this case study. Only the role of the EACEA in the accompaniment of the projects is sometimes unclear. Once a project is running, it communicates directly with EACEA about the progress in its implementation. The EACEA strives to have close contact with projects. However, it is sometimes not clear to beneficiaries whether the role of the EACEA is purely administrative or also targeted at the content of projects. The EACEA has built up a lot of ground-level expertise through its close contact with projects during the implementation, however its formal responsibilities are administrative and managerial. The EACEA defines its role very strictly as focusing mainly on financial and managerial issues; however, beneficiaries sometimes have different expectations, for example expecting still closer guidance on the content of projects. The EACEA visits projects that are in difficulties and those that can be considered good practices. Not all projects are visited. Some of the beneficiaries that were interviewed for this case study recognise a much closer and more active involvement by the agency than what they were used to from the pre-2007 technical assistance office whereas others perceive the agency as rather remote. Regarding the low number of interviewees, these perceptions should however be treated with care. The reporting and monitoring stage. The monitoring of projects and the reporting is done very diligently and to the deadlines set by the framework. Projects provide a lot of information and get a lot of feedback from the EACEA. However, the information collected could be used more effectively. The monitoring framework stipulates that projects have to provide monitoring information to the EACEA in the form of a progress report and a final report. The EACEA uses the progress report to evaluate whether a project faces problems and needs closer attention. After a project is finished, it gets evaluated by the EACEA and extensive feedback is sent to the beneficiaries. In general terms this monitoring and reporting framework is working well. Beneficiaries interviewed appreciate the feedback, though some find the burden of reporting too high. The EACEA also gets enough information in order to be able to respond to problems or, if there is no way out, terminate projects. However, the information collected could be used much more extensively in order to improve the programme. At the moment, any final evaluation of a project that receives more than 80% of the points is sent to the Commission unit on multilingualism as a good practice. However, no additional information is provided and no analysis of the results is done. This means that a lot of information is lost which could be 97

98 used by the Commission to adjust the programme according to the lessons learned by the projects. The officials interviewed expressed a strong need for more analysis of the monitoring information. For this analysis to take place, more human resource capacities are still needed. 4.2 To what extent is the implementation and management structure of the KA2 and other relevant actions efficient and well functioning i.e. funds reach the intended target groups; are disbursed with no delays; with no systemic irregularities? Could the KA2 be effectively decentralised with better reach of target groups less administrative deficiencies and no loss of European visibility and dimension (and vice versa)? The implementation and management structure of the KA2 works well. The work is carried out to the deadlines agreed. The human resources available at the Commission and the EACEA are sufficient for the work that is done at the moment. An increase in the workload would lead to a need for new resources. The statistical reports show and the interviews with officials confirm that the available budget of the KA2 is disbursed almost entirely to projects. Only in one year, some money was given back to the programme since it was not enough to fund yet another project. According to the information collected in the interviews, one project was terminated because it could be foreseen that it would have had to pay back most of the funds. This was prevented by not paying out the funds in the first place. There is no need to decentralise the KA2. Due to its transversal and supra-national character, it is rightly managed centrally on a European level. However, when looking at the implementation of the horizontal priority of multilingualism in general, some issues arise regarding synergy and integration of the programme. As said before, language activities are taking place both at a centralised and at a decentralised level, in subprogrammes as well as in the Transversal programme. The explicit objective of the KA2 is to complement the programme in providing for large-scale, cross-sectoral projects. However, interviews carried out for other case studies and the national reports indicate that National Agencies would like to be better informed of the activities carried out at a centralised level, regarding the selection of projects and the results achieved by projects. In general, little exchange takes place between the EU level and the national agencies. While the levels of management of KA2 and sub-programme activities correspond to their role in the overall programme, more cooperation (horizontally and vertically) can lead to still more exploitation of the benefits of integration of the programme. Regarding the sectoral sub-programmes, as far as the horizontal policy of multilingualism is concerned, no issues regarding centralisation or decentralisation of actions arise. 4.3 How effective is the cooperation between the different management bodies (the Commission the Executive Agency the National Agencies national authorities the LLP Committee)? To what extent does the Commission fulfil its guiding role in the process? For the KA2, the cooperation between the EACEA and the Commission needs some attention. The different responsibilities are not clear to all beneficiaries and still more benefits of cooperation between the mirror units of Commission and EACEA can be exploited, especially regarding the sharing of results achieved by projects. Regarding the implementation of the horizontal policy in general, the cooperation between the policy unit responsible for multilingualism and those units responsible for the sectoral sub-programmes can still be intensified in order to assign higher priority to multilingualism and linguistic diversity throughout the programme. Though the EACEA duly fulfils its role as a managing and administrative body, in its daily work, it also automatically advises projects about the content of their work. As a consequence, it builds up a lot of experiential expertise about the needs of the target beneficiaries, the problems and success factors during implementation of projects and similar issues. This expertise can be useful in the setting of priorities or writing of Calls for Proposals. The sharing of expertise between Commission and the EACEA can be increased in order to make more use of the expertise built up by the EACEA. The knowledge and sensitivity for the practical issues that the EACEA has built up, need to be exploited by the Commission when drawing up new priorities and calls for proposals. 98

99 Looking at the entire programme, the horizontal policy could be given even more shape by establishing more intensive cooperation and exchange between the different units and officials dealing with language issues. According to the officials interviewed, where necessary this cooperation already takes place. However, more opportunities exist, especially on a strategic level. It is not only important that the sectoral sub-programmes report their relevant results to the multilingualism policy to the unit in charge of that policy. It would also be helpful if the officials in the sectoral sub-programmes were more aware of the projects carried out in other sectors and in the KA2. By cooperating and exchanging information especially during the evaluation phase of projects, lessons can be learned and new synergies can be created. This way the horizontal priority of multilingualism can be strengthened throughout the entire programme. 99

100 4. Jean Monnet KA1 Excellence Centres and Chairs 1. INTRODUCTION The overall purpose of this case study is to provide an in depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the Jean Monnet Action, focusing on aspects of both Action (a) stimulating teaching, research and reflection in the field of European integration studies at the level of higher education institutions (specifically Jean Monnet Chairs) and (b) Other academic and research projects (specifically Jean Monnet Centres of Excellence). This is KA1 of the Action, Centres of Excellence and Chairs. The case study provides a general overview of the work of the programme within an overall European context, and then focuses on the UK where evidence is gathered from two sites, one at the University of Edinburgh, through a Jean Monnet Chair and the second at the London School of Economics through a Jean Monnet Centre of Excellence within which a chair is embedded. The information presented in this case study is based on the following sources: - In-depth interviews with two officials at the EC (two interviews) and the institutional beneficiaries of the Jean Monnet action (four interviews). The full list of interviews is presented in the Annexes; - Desk research, which sought to analyse national consultation papers, Green Papers, White Papers and legal acts, the action s implementation statistics at national and EU level, and a range of other documentation. Data was provided by the EC and the institutional beneficiaries. Other documentation was obtained through web searches and through traditional academic sources. This included material from relevant agencies in the UK concerned with Higher Education, including the Higher Education Funding Council for England (HEFCE), the Higher Education Academy (HEA), Centres for Excellence in Teaching and Learning (CETLs) and the Scottish Funding Council (SFC). It should be noted that in the UK, HE funding is devolved to the Member States, and as a consequence national policies, which may complement those of Jean Monnet, vary across England, Scotland, Wales and Northern Ireland. Since the two institutional beneficiaries are in England and Scotland, only specific documentation pertaining to these countries has been sourced as well as UK wide material. Material from National Agencies concerned with LLL in the UK was also consulted in order to assess synergetic and overlapping effects. The interviews and desk research were carried out during the period of 1 May to 16 September RELEVANCE The main objective of this section is to describe the European Added Value of the selected action To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the national policy priorities? The specific objectives of the Jean Monnet sub-programme, as set out in Article 35.1 of the sub-programme decision, are: (a) to stimulate teaching, research and reflection activities in the field of European integration studies; (b) to support the existence of an appropriate range of institutions and associations focusing on issues relating to European integration and on education and training in a European perspective. The operational objectives of the Jean Monnet sub-programme, as set out in Article 35.2 of the sub-programme Decision, are: (a) to stimulate excellence in teaching, research and reflection in European integration studies in higher education institutions within and outside the EU; (b) to enhance knowledge and awareness among specialist academics and among European citizens generally of issues relating to European integration; (c) to support key European institutions dealing with issues relating to European integration; (d) to support the existence of high-quality European institutions and associations active in the fields of education and training (see EC (2008) and EC ( ) 44. According to our findings, the Actions of the Jean Monnet subprogramme appear to be congruent with UK national policies concerning lifelong learning, particularly those related to the international competitiveness agenda. However, the Jean Monnet KA1 is seldom if at all mentioned in most of the UK national policy documents related to lifelong learning, skills and the role of 44 EC (2008) The European Union and the World European Success Stories. Brussels: DGE&C EC (2007-9) Jean Monnet Programme Statistics. Brussels: EACEA. 100

101 higher education or within UK analytical documents related to Higher Education and Europe that have been consulted. In order to contextualise the UK situation, it should be noted that the country has experienced a surge of immigration from Eastern Europe following the expansion of the EU in 2004 (especially young adults under 40 from Poland), and brought to the fore issues pertinent to the Jean Monnet KA1, most specifically operational objective (b) above. EU cohesion has become a major focus of the Government policy, fed particularly by concerns about the social exclusion of some ethnic and religious minority groups. There has been a growing attention to the integration of new migrants. Within the UK national policy that pertains to Jean Monnet is elucidated by a number of bodies. Since the mid 1990s there has been a succession of consultation documents, Green Papers, White Papers and Acts of Parliament, which have focused on aspects of lifelong learning, including the role of Higher Education, in particular its part in ensuring international competitiveness (see for example Fryer 1997, DfEE 1998, Leitch 2006 and DIUS 2007). 45 HEFCE, which provides the bulk of public funding to the HE sector in England states quite clearly that Competing within the global HE market and developing international links is primarily a matter for universities and colleges themselves, but we have a strategic role, along with other key stakeholders, to support institutions in promoting a distinctive high-quality 'brand' for UK higher education (HE) and to maintain its international competitiveness. 46 There also exists a Europe Unit funded by HEFCE and funding councils in other parts of the UK, and by Universities UK (the organization representing Vice- Chancellors). This aims to raise awareness of the European issues affecting UK higher education and to coordinate the UK s involvement in European initiatives and debates and to strengthen the position of the UK higher education sector in debates on the Bologna Process and EU policy. It is also notable that the UK seeks to strike a balance between cooperation and competition with other members of the EU with regard to matters pertaining to HE. It supports aspects of integration and harmonisation such as those laid out in the Bologna agreement of which the UK was an original signatory in 1999, but also recognises that there is competition from other countries in the EU, particularly for international students. However, even in this area of competition it is interested in collaboration as well (see UK Higher Education Unit and UK International Unit (2009) 47 ). One might also expect some congruence with a number of the challenges highlighted in UK Progress Reports on meeting the Lisbon Objectives (2005, 2007 and 2009) 48, but it is difficult to ascertain such statements in these documents since no specific reference to the JM programme exists. To a certain extent this is to be expected given the comments from one of our respondents at the EC. She observed that the target of the scheme was not national governments but universities themselves. The objective was to select excellent individual professors and establish excellent centres that would offer a critical perspective and add to the debate about European integration. She supports the notion that the phenomenon is studied with its weaknesses and strengths, and that those funded are not simply ambassadors of the EU, but high-level independent scholars. 2.2 To what extent do the intermediate and specific objectives of the action remain pertinent to the lifelong learning needs of the target groups? The findings suggest that the objectives of the action are somewhat pertinent to the needs of target groups, yet the latter are rather narrow. The principal target groups of the JM programme are institutions and their staff. Demand for funding and the high quality of applications are indicators that institutional beneficiaries are being reached. From the perspective of a JM activity administrator, this demand demonstrates continued 45 Fryer, R. (1997) Learning for the Twenty-First Century: First Report of the National Advisory Group for Continuing Education and Lifelong Learning, DfEE (1998) The Learning Age: a renaissance for a new Britain, London: The Stationery Office. Leitch, S. (2006) Prosperity for all in the global economy- world class skills, DIUS (2007) World Class Skills: Implementing the Leitch Review of Skills in England, 46 Higher Education Funding Council for England, International Section 47 UK Higher Education Unit and UK International Unit (2009) UK Universities and Europe: Competition and Internationalisation. London: UK Higher Education Unit UK Higher Education Europe Unit UK Higher Education International Unit 48 UK Progress Report towards the Lisbon Objectives

102 relevance and success of the programme in a number of ways. For the EU27 (which still provides majority of applications) this included exploring issues of constitutionalism, implications of structural changes and voting behaviour, all important aspects of European integration. For candidate countries, the action is very important as JM professors are often heavily involved in providing knowledge in these countries and in assisting with negotiation processes. For the Rest of World, the action assists in understanding of the EU s political, economic and legal processes. For individual applicants, JM title confers prestige, and provides important networking opportunities. A respondent from the EC emphasised how important the brand of Jean Monnet is. More generally the target groups of Jean Monnet are in addition to HEIs, academics and students are associations and members of the public in general. Hence there are direct beneficiaries at institutional level (i.e. the HEIs) and individual level (i.e. staff and students of those HEIs) and indirect beneficiaries within the wider community that the Programme serves. It can be argued that academics just like other professionals have professional development needs, and activities with the programme including public lectures, conferences and discussion groups with parliamentarians that were highlighted by institutional beneficiaries clearly offer informal opportunities for lifelong learning. These provide direct reputational benefits to HEIs and their staff through highlighting their role externally as centres of excellence in relation to European matters and through creating various networking opportunities. Opportunities afforded to HE students as direct beneficiaries both at undergraduate and postgraduate level expand knowledge and competence in European Integration matters and in particular contribute to employability. Outreach activities provided by the direct beneficiaries also can contribute to lifelong learning within schools, colleges, workplace and a variety of settings, hence create a number of indirect beneficiaries within the communities served by HEIs, as was reported by one of the beneficiaries who was interviewed. In-reach activities, (i.e. providing access to otherwise daunting institutions) also create indirect beneficiaries in communities. Events such as public lectures provide opportunities for informal learning to the general public (e.g. at the Jean Monnet Centre of Excellence, The idea of Europe: concepts and emerging practices, at the London School of Economics) and may contribute to social inclusion. It should, however, be noted that when asked to quantify this, the coordinator interviewed said this was not possible as data was not collected on the composition of the audiences. He said that gathering such information from the individual members of the audiences would be considered intrusive and counter-productive. The main objective is to offer up-to-date lectures on EU-related issues with significant speakers and to publicise them to the public as widely as possible. It is interesting to note that in a previous evaluation of JM (ECOTEC 2007) 49, with only 15% of 162 respondents holding awards believing that participating in Jean Monnet actions had developed links with the general public to a large extent. This was felt at this time by the evaluators not to surprising, given that the focus on the most frequent Jean Monnet activities (Modules and Chairs) is primarily on academic target audiences (students and fellow academics) (ECOTEC, p. 160). These researchers also reported that In general, interviewees felt that efforts within the Centres of Excellence in which they had been involved to promote debate and information sharing with the general public had been among the least successful element of their activities. This is almost certainly because working with the general public requires different approaches and skills from those typically needed when dealing with an academic or practitioner audience. It would appear that the interviewees in this case study had had more success in this regard. It is also notable as reported by a respondent at the EC that the JM Unit places importance on respecting academic independence and that it does not seek to orientate the emphasis of those it funded. However, there is strong advice to have an impact on general society, and she estimates that some 80% of Centres of Excellence are strongly linked to external bodies. 2.3 The European added value. Would other national or international schemes / instruments provide enough support to related activities, if there was no JM KA1? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? Our findings cannot provide evidence if similar level of academic activities would take place without funding under Jean Monnet KA1. However, it is clear that there are strong inner incentives within the selected institutions to seek highest academic quality and visibility. Funding from the LLP is therefore viewed as one of the alternatives to secure funding for relevant activities. 49 ECOTEC (2007) Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training Final Report. Birmingham/Brussels: ECOTEC 102

103 In the UK there are a range of bodies concerned with aspects of lifelong learning that are pertinent to the Jean Monnet Action, but most tangentially so. The Higher Education Academy, supported by the UK Funding Councils, seeks to support the higher education sector by working with and has a network of 24 disciplinebased subject centres that provide a range of services to academic discipline areas. The HEA also provides national leadership in developing and disseminating evidence-informed practice about enhancing the student learning experience. HEFCE has also funded 74 Centres for Excellence in Teaching and Learning (CETL) 50 initiative with two main aims: to reward excellent teaching practice, and to further invest in that practice so that CETLs funding delivers substantial benefits to students, teachers and institutions. Lifelong Learning UK is the independent employer-led sector skills council responsible for the professional development of staff working in the UK lifelong learning sector, including in HE. A search of the work of the HEA 51, CETL and LLL UK 52 (the Skills Sector organization for lifelong learning) indicates little that refers specifically to either the development of staff or of students in relation to the issue of European Integration as covered by the operational objectives of the four operational objectives of the Jean Monnet Action. Mention of the action at all is seldom at all found. Many HEIs offer programmes within both undergraduate, post-graduate and continuing education programmes (albeit substantially reduced for the general public interest from previous decades because of funding changes) that relate to these issues, including the two institutions subject to detailed analysis in this case study, and Jean Monnet funding both supports existing activities and creates opportunities that would not otherwise exist. UK universities also are charged with a public engagement role with a national cocoordinating centre, the National Beacon for Public Engagement, stimulating this activity. 53 This area of work offers the potential for synergies with operational objective b) of Jean Monnet, although there does not appear to be a national project focusing on engaging the public in matters pertaining to European Integration. Evidence at the level of individual beneficiaries in the UK (see respondent in section 3.1) indicates clearly that developments stem from the JM Action that would not otherwise occur. Our respondent summarises a number of the benefits from his perspective. These include: strengthening of teaching on the EU at the University; links with study groups and networks on law and political economy; enhancing the learning environment for students, at both undergraduate and postgraduate levels; and facilitating community involvement. This reflects one of the priorities of the project, Priority 3 of the JM Actions, covering teaching at undergraduate and graduate level and supervision of research at postgraduate level. Our interviewee at the Centre of Excellence at the LSE reports that they already run an extensive programme of public lectures. These are the most extensive in Europe outside of Bruges with very high profile speakers. Whilst at one point the Institute did not need JM status and have been meeting their internal objectives, there is ongoing pressure on resources so the JM Action is an important alternative means of support. Furthermore, the European Institute has to purchase services from other departments so they always face out of pocket expenses, and here JM funding is helpful. It can be concluded that structurally and organisationally the UK has the capacity to create synergetic effects through a variety of organisations, but that without Jean Monnet, relatively little coordinated activity would occur. At institutional level the JM action both lets activities occur that otherwise would not, and reinforces activity that already exists. The extent of JM funding is relatively small, and it may be concluded that a focus on institutions where there is already a base of significant activity created most effect. Our respondent from the EC commented that, with the main criteria of JM being quality and potential impact, building on existing strengths was indeed important What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Many funded JM projects contribute to multiple LLP Learning Objectives, and some to almost all. It therefore can be expected that there are synergies and overlaps with other sub-programmes. These potential synergetic effects and overlap between the Jean Monnet Actions and other sub-programmes might be 50 See Higher Education Funding Council for England Centres of Excellence for Teaching and Learning,, 51 See Higher Education Academy at 52 See Lifelong Learning UK at 53 See National Beacon for Public Engagement at 103

104 expected with the Erasmus Intensive Programmes, Academic and Structural Networks and Multi-lateral projects, given the objectives of these fields of the LLP. This is evidenced as a reality through a search of the Compendium of funded Erasmus projects in 2008 for example finds a Multi-lateral Project ( LLP ES-ERASMUS-ECDSP) and an Erasmus Network ( LLP IT-ERASMUS-ENW), which explicitly focus on European Integration. However, it is not obvious as to whether synergetic effects have been explored, because the interviews show there is little awareness of the different sub-programmes and actions among people working with the activities funded by JM KA1. Overall there seems little explicit duplication with Erasmus, but perhaps synergies that could be exploited. In this context, the respondent from the EC commented that clearly JM was a congruent with the aims of the overall LLP, being an aspect of HE work. She suggested that it adds a different dimension to the LLP by offering a knowledge domain to Erasmus sub-programme, which focuses on mobility. Her experience is that there are within certain institutions links between JM and Erasmus, and she knows of many cases where Erasmus coordinators ask JM CoEs or professors to offer courses to visiting students about European integration. However, she agreed there could be stronger synergies at institutional level with Erasmus, including Erasmus Mundus. For example, in all Erasmus Mundus Masters programmes there could be a component on integration studies. There could also be links with the Tempus and Marie Curie programme. At many institutions, however, it is quite likely that the work of JM and that of other parts of the LLP occur in quite distinct parts of organisations. For example, one interviewee at the LSE explained that Erasmus and Jean Monnet funding had completely different functions in his organization and there was no overlap whatsoever. There are of course potential synergies with the Transversal programme, particularly KA1 studies with a research focus, project in KA2 concerned with language learning and intercultural understanding and those in KA4 concerned with universities and civic engagement. Yet there is no evidence from documentary evidence and interviewees were not able to provide any concrete examples. It is important to stress that institutional beneficiaries had little knowledge of other aspects of the LLP. A further area of potential overlaps occurs across the various Jean Monnet actions: Jean Monnet Chairs, Centres of Excellence and Multilateral Research Groups. These could occur because of similar target groups: only the institutions with Jean Monnet Chairs are eligible to apply for Centres of Excellence and Multilateral Research Groups. Therefore all three actions provide support to Jean Monnet Chairs. However, despite overlapping target groups, the Jean Monnet actions provide support to different types of activities. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved. 3.1 Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? What are the main external factors affecting the outputs and results of the action? According to EC (2008) statistics, the action is present in 61 countries on five continents. Between 1990 and 2008, the Action has helped to set up approximately teaching projects in the field of European integration studies, including 134 Jean Monnet European Centres of Excellence, 798 Jean Monnet Chairs and European modules and permanent courses. The Jean Monnet Action brings together a network of professors, reaching audiences of students every year (though it should be noted that contradictory statistics are presented at the LLP website). In 2007, 11 Centres of Excellence from 15 applications, 34 chairs from 76 applications and 7 chairs (ad personam) were awarded. The equivalent figures for 2008 were 10 from 17, 27 from 58 and 10 from 23. The Centres of Excellence have higher than average success rates compared to all JM Actions. In the UK, the specific country of this case study, two chairs were appointed in 2007 and three (from five applications) in 2008, and in the latter year a Centre of Excellence was also awarded from the only application. Success rates in 2008 for the UK are amongst the highest of all applicant countries across JM Actions at 67%. In 2009, four 104

105 chairs and two Centres of Excellence were awarded in the UK. In response to question of progress in achieving the targets of the Jean Monnet Action, one of our respondents reported the view that Jean Monnet is an important programme for Higher Education to contribute to key EU objectives, especially: promoting inter-cultural dialogue; explaining the EU, and making a bridge to citizens; and understanding and explaining the EU and its processes within the EU and particularly outside the EU, where the JM programme now reaches 62 countries. From his perspective, the JM actions stimulate excellence in HEIs in teaching and research about the EU. He also reported that of all the centralised actions, JM KA1 is the most successful in terms of the number of applications. It has attracted a steady increase in applications, and now requires the highest assessment scores for success, compared with other programmes. The quality of proposals accepted is outstanding, and in 2009 only 24% of applications could be accepted within present budgetary limits. The survey of beneficiaries showed that the share of very satisfied individual beneficiaries to be high all the sub-programmes (around 70%), but the share of very satisfied institutional beneficiaries especially the percentage of participants of the centralised actions such as JM was notably lower. However, external factors such as mobility barriers, language barriers, financial position of organisations and individuals, lack of support from potential socio-economic partners, lack of capacity or willingness of potential beneficiaries, unfavourable personal values and attitudes of potential participants and changes in national educational policy are all notably lower in the JM Action than in any other sub-programme. None of these factors is above 30%. A beneficiary at the LSE added weight to this finding saying that there were no external obstacles that could be identified or factors affecting the outputs and results of LSE s work. At the level of individual beneficiaries, the Jean Monnet Chair at the University of Edinburgh, who works on the political economy of EU Integration reported particular additional developments that were supported, which would not have been possible without the JM award. These included: a visiting speaker series of events; additional teaching assistants, which has allowed additional courses to be offered; outreach to a wider community, including schools, practitioners with an interest in EU, institutions such as banks and government; and organisation of major conference on EU finance. At the LSE, also it has been reported the particular value of JM funding. Our respondent suggested that it is difficult to get finance in the European Studies area for smaller projects. Most project funding seems to be for large international projects whereas at LSE s EU Institute they undertake a lot of highly specialised development work. The same applies to their researchers looking for funding. So the scale of JM funding is very helpful, welcome and more accessible than other avenues. As also indicated by another respondent it can make a real difference in institutions where there is already a significant strength in the field of European integration. 3.2 Effectiveness in implementing horizontal policies: How does the action contribute to the horizontal priority of multilingualism? How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? It is difficult to ascertain whether the JM Action contributes to the horizontal priority of multilingualism or to equal opportunities. Clearly JM promotes inter-cultural dialogue, but as one individual respondent replied, multilingualism is not part of the impact assessment. The openness of public events arranged by both sets of beneficiaries interviewed, indicates that respect to equal opportunities is provided at this level, although monitoring of projects cannot give definitive evidence that this is a principal concern. Further at the LSE, our respondent reports that JM funds are specifically being used for one target group, namely signing or help for visually impaired in public lectures. Outreach activity to schools such as that of the University of Edinburgh also provides some evidence of attempts to reach new groups, but equal opportunities does not appear as a principal activity of JM. In the UK links between schools and HE are part of a major concern of education policy in the country, to raise awareness of HE possibilities to pupils from disadvantaged backgrounds. Widening participation to HE is signalled in reports of the National Agency for the LLP to the EC as a principal concern within decentralised 105

106 programmes, and it is also part of national policy imperatives in the UK. Links via the JM Action could play a part in raising awareness of HE generally as a by-product of raising awareness of European integration. At the LSE, a respondent reports that widening access to the university is something that it does anyway and is taken seriously, and they already are involved in a range of outreach activity, but this is not a focus for the JM work. In the UK the wider issues of access are already something that most universities are trying to address. There is evidence that JM projects are tangentially also making a contribution though this is not always systematically monitored or linked to nationally funded initiatives in the domain of widening access either specifically to undergraduate education or more generally to the initiatives for wider public engagement. There would seem to be opportunities for development in this area Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? There is little evidence of substantial integration across sub-programmes from the JM Action, even though potential benefits of integration for HEIs have been identified. As with other parts of the LLP, there was limited awareness of other good practices and project results amongst beneficiaries who responded to the survey. This was also the case amongst some of the interviewees. For example at the LSE, the only other programme they take part in is Erasmus which in terms of mobility is in some way superfluous, as the LSE receives many international students anyway. Our respondents reported that there was no intentional working between Erasmus and JM in place. Hence they cannot obtain and actually are not in need of EU funding at the LSE for mobility of students. For the LSE, what is important is to be able to make a distinction between research and teaching and JM allows them to do that Effectiveness in reaching the target groups and disseminating results: How successful is the action in reaching the target groups and institutional beneficiaries? To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels 54 )? At the level of reaching institutional target groups, statistical evidence produced by the EACEA 55 indicates overall a high level of success in the objective of achieving an international coverage in terms of beneficiaries, albeit with a fairly static (in some cases declining) number of applications and funded bids in certain areas. Overall within all actions of JM, there was in 2008 an increase of the total number of countries participating to the programme with a proposals selected from 61 countries from all over the world). More specifically in relation to the focus of this case study, namely Centres of Excellence and Chair, the statistics are as follows. For Jean Monnet Centres of Excellence in 2007 there were 15 applications and 12 selections, whilst in 2008, those numbers were 17 and 10. For Jean Monnet Chairs in 2007, there were 76 applications and 34 selections, whilst in 2008 these numbers reduced to 58 and 27. For Jean Monnet Chairs "ad personam", there were 21 applications and 7 selections in 2007, whilst in 2008, these numbers were 23 and 10. Of the 41 Chairs and Chairs "ad personam selected in 2007, 29 were awarded within EU countries, 7 to other European countries (Belarus, Bulgaria, Macedonia, Romania and Turkey (3) and 5 outside Europe (Brazil, Canada, China, Mexico and South Korea). Of the 12 Centres of Excellence selected in 2008, 9 were in the EC, one elsewhere in Europe (Turkey) and two outside Europe (Japan and the US). Of the 37 chairs and Chairs "ad personam selected in 2007, 28 were awarded within EU countries, 5 to other European countries (Croatia (2), Georgia, Iceland and Turkey and 3 outside Europe (Canada (2) and China). Of the 10 Centres of Excellence, all were in the EC. 54 Horizontal mainstreaming refers to the project results and good practices adopted by other stakeholders. Vertical mainstreaming refers to the integration of the good practices and project results into general policy on national and/or European level. 55 EACEA (2008) Results of the 2008 LLP Call for Proposals. Jean Monnet Programme Statistics. Brussels:EACEA 106

107 In 2009, of the 38 Chairs and Chairs "ad personam selected, none were awarded to European countries, outside the EU, and 7 to non-european countries (Australia, Canada, China, Japan and US (3). Of 11 Centres of Excellence, 4 were outside the EU (Argentina, New Zealand, Romania and Russia). This would indicate moderate success in relation to the objective of reaching institutions outside the EU, although it is largely a function (based on 2008 statistics) of the numbers of applications received. Many eligible countries both within and outside the EU simply did not make applications. Where they did largely their rates of success were comparable. For example, from outside the EC, Canada made 6 applications and received 3 acceptances in 2008 for chairs or chairs ad personam. Some countries simply made no applications for chairs or Centres of Excellence. Other countries by comparison have poorer success rates, but make many more applications, and overall have higher absolute successes. Notably in this category is Italy with 13 applications for chairs and 5 successes, and 5 applications for centres and 2 successes in Italy also was notably funded for 8 chairs in Also notable is the success of Spain in 2008 with 9 funded chairs from 13 applications and 3 funded Centres from 4 applications, and the Netherlands with 4 applications for chairs, all successful. It is also notable that only two countries from outside the EU made applications for Centres of Excellence in 2008, both unsuccessful (Canada and Croatia). When more applications were made in 2009, the coverage became more extensive. One may therefore conclude that although across all actions of JM, there are a large number of beneficiary countries, this narrows within Chairs, and even more so in Chairs ad personam and Centres of Excellence. Taking 2008 as an example, only 7 EU and 6 non-eu countries benefited from Chairs, 5 EU and 1 non-eu from Chairs ad personam, and 7 EU countries from Centres of Excellence. As indicated above to a large extent this is a function of numbers of applications, and raises the question of whether demand is being stimulated sufficient both within and outside the EU, and the extent of knowledge that exists of these parts of the JM programme. Data for 2009, however, shows an overall improvement in the range of beneficiary countries for chairs with 15 EU countries and 5 non-european countries benefiting from Chairs or Chairs ad personam. There is a narrowing of EC beneficiaries for Centres of Excellence with only 4 EC countries in receipt of funding, but one other European country and three non-european countries are funded. As in previous years there are concentrations of success with in 2009, Spain winning 3 of these centres and 5 Chairs ad personam. Similarly, France won 4 Chairs, and this phenomenon is worthy of further analysis. In this context, a respondent from the EC emphasised that there were no quotas and no attempt to seek balance by country, reiterating her view that the focus is on the best and brightest, and on institutions rather than nations as beneficiaries. At the level of the cases under study in UK, evidence is provided of the various ways in which there is dissemination to indirect beneficiaries. For example, our respondent from the Centre of Excellence at LSE reported that they monitor attendance at lectures and when they do things for charities. However, whilst they hold open events they do not know what groups come and how much they benefit. Since it would be very difficult to monitor such events for particular target groups, this centre cannot report as to whether they get members of socially and economically disadvantaged groups and if they do it will be few. In this case it is possible a function of being in central London, and issues about transport and mobility from areas where such target groups are likely to live. However, other specific target groups including diplomats, European NGOs and individuals from research centres are reported as being part of their audiences for events. Sometimes public events are picked up by the media, including journalists from the quality press, but this is not something that the Centre currently seeks. The Centre also publishes working papers, uses websites and podcasts set lectures, thus adding impact to these events which usually attract 150 people in person. Our respondent also reports that the European Centre also have specialist smaller seminar round table events, smaller events with experts and practitioners and academics from 6-8 in the evening, and a series of events for between 30 and 70. The JM funding primarily will be targeted at the small bespoke events, extending work already developed. These events can be expected to happen regardless of JM funding, yet they are currently an integral part of dissemination efforts within JM. Whilst some activities are expected to conclude with the project and others will continue, for example, further development of web site and the public lectures. The institution intends to build on the results of the programme, thus making ongoing sustainability an issue of intention. The centre hopes to continue to revisit topics, for example, review of the European budget which is ongoing. It sees their events as part of a continuum of debate and discussion. Our respondent did not identify any external factors affecting the progress, and noted that the centre are on 107

108 target for what they want to achieve with the programme, including the number of specific actions, their public lecture series and a research project on the single market amongst others. Another beneficiary at the University of Edinburgh reported being confident in reaching targets for activity and benefit to academics and students. Students are generally very enthusiastic and greatly benefit from involvement input from EU officials, and from immediacy of considering issues such as the sovereign debt crisis. However it was more hit and miss with involvement of the wider community, into which considerable effort has been put. This was even the case in a University with a strong outreach ethos/policy. Like the LSE case events were successful in reaching a number of audiences. A conference was very successful in attracting Members of the European Parliament (MEPs), Members of the Scottish Parliament (MSPs) and people from a number of relevant institutions. Our interviewee writes on the EU single market, and is not sure how much policy-makers and politicians read. It may therefore be important for them to attend events in lieu of this. Audiences for his other public events have varied, depending on topic, but these events have provided an opportunity for networking between stakeholders, which would not have been there otherwise. Our interviewee personally undertakes two or three school visits per year, and more requests are being received annually from schools. He has also developed other communication methods such as podcasts to increase accessibility of public events. The JM programme gives greater credibility and recognition in academic circles. Our respondent considers the increased status and award as a Centre of Excellence would enhance what can be achieved and attract increased attention. In addition, outputs are fed back to the EU policy process in a number of ways. These are mainly through those EU conferences that invite JM professors and centres to contribute to dialogue, and to feed directly into Council of Ministers. This is a tradition of the JM Programme, namely of consulting the academic world in order to reflect on current issues of European integration, bringing together policymakers, civil society and top-level academics. Recent examples are: - The Jean Monnet conference on the role of education and training in the EU 2020 strategy which was organised at the explicit request of the Spanish Presidency in January 2010 to feed into the EU 2020 policy process (European Council meeting on EU 2020); - The Jean Monnet conference on the impact of the Lisbon Treaty (in May 2010). This event constituted a structured dialogue between Commissioners, Members of the European Parliament, journalists and Jean Monnet professors; - The 2008 Year of inter-cultural dialogue was the result of a series of Jean Monnet conferences and reflection activities organised by the Jean Monnet Unit, with the participation of the Jean Monnet professors, since The Gonzalez Report on the EU in 2030 explicitly expresses thanks for the input of the Jean Monnet community. This input was organised following a call made by Prof. Nicolaidis, a member of the Gonzalez Reflection Group, at the 2009 Jean Monnet Conference. It is the view of the EC that the independent and critical considerations of the members of the Jean Monnet network have provided the policy-makers with new insights and concrete policy suggestions. There is some evidence at the level of individual beneficiary that arrangements for attendance at conferences organised by the EC could be improved by introducing more incentives to attend. Sometimes notices have been short and without funding support. With integration spreading in more fields (e.g. services, company law etc), there is an enhanced need for well-trained lawyers, economists and other university graduates who can make the new single market work. Ensuring that university graduates get the skills and knowledge to use the tools of European integration is exactly what the Jean Monnet professors are doing throughout Europe. Our respondents add that the JM programme continues to increase the visibility of the EU across the world. There is a large demand for information and analysis on the EU in third countries. Jean Monnet professors provide this on the five continents and in a decentralised manner (spread at universities all over a country, not only in the capitals where one also finds the EU delegations). The subject matter of JM KA1 EU integration requires academic multi-disciplinary and cross-country expertise. The JM model is being copied by such bodies as the World Trade Organisation in Geneva. The accumulated knowledge-base is enabling the EU to help itself in a way which is independent of national concerns, and allows critical open method of working. The JM programme increases the process of critical reflection. Further, JM project activity reports, books and articles are all deposited in the EU library and accessed by Commission, and picked up by relevant directorates on completion. Lists are circulated on a daily basis. 108

109 Conference reports are also made available to the public via the Jean Monnet website and in hard copy form. A respondent representing the beneficiary, however, also spoke of ways of improving effectiveness in dissemination. He offered several suggestions: - The Jean Monnet network could be used as an even more powerful think tank for the EU institutions. For this purpose, it would be very useful to create a more developed and interactive project directory (indicating expertise and relevant publications) of Jean Monnet professors; - It might be ensured that all pertinent Commission services are alerted to relevant project activity reports. In this context, EACEA has been already asked to prepare project summaries on relevant Jean Monnet research and debate projects; - Some Commission officials have contacted the JM Unit to inquire whether attendance of Jean Monnet conferences could be included in their training plan. In view of the high relevance for Commission staff of the topics discussed by Jean Monnet professors, it might be useful to organise regular seminars on topical issues over lunch with prominent Jean Monnet professors as speakers for the benefit of the entire Commission staff (this could then be part of the training plan of those colleagues). It would appear clear that JM funding within the two universities studied makes a difference and provides added value in a number of ways. There would seem to be merit in focusing the relatively small funds of JM to places where an existing infrastructure exists. The previous ECOTEC (2007, p. 164) study has reported that the Action achieves a particularly impressive range of results and impacts with comparatively little money, and the range of dissemination activities reported here does indicate for the two cases that this also appears to be the case in these instances. However, at institutional level whilst there are various forms of dissemination, these are not always obvious in electronic form. It was not obvious that at the level of Jean Monnet Professor or of Jean Monnet Centre of Excellence from the sites of our interviews that substantial profile was given electronically to the work funded from the EC at an individual level. That is not to say that this does not happen, but the work of an individual Jean Monnet professor tends to be subsumed into that of a larger unit, and perhaps does not get enough discrete exposure. Examination of available website materials relating to JM at the EC indicates relatively little information at individual project level, when the database is searched. The main dissemination mechanism to a wider public through the JM site appears to be the Compendia of Success Stories of 2007 and 2008 within which short profiles are found, and conference reports. The EC representative interviewed agreed that dissemination at individual university level could be improved, but this was a resource issue. She suggested that some universities are better at promoting the work than others, but the JM unit was not directive in this regard and it had avoided promoting good practice. Some individual ad personam professors promote themselves extensively (a professor in New York was cited) as do some CoEs (that in Rennes 1 and 2 was cited). Since ad personam professors retain the title, some promoted JM by transferring the title when they move. Our respondent also cited a UK professor who had taken the title to Yale. An important aspect of the overall success of initiatives is sustainability once funding disappears, and the institutions in this case study provide good evidence of forward planning with this regard. Overall the relative cost of the scheme in relation to the number of projects funded indicates good value for money. In the view of three of the respondents, it may be considered as a best practice (cost/effective activity ). 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? At the level of beneficiaries, the survey to every unique within JM (with a response rate of 17%) indicated a remarkably high level of satisfaction in relation to key aspects of management. Together with 109

110 KA1, the JM programme is perceived most positively with regard to aspects of management with many items receiving a satisfaction rating of over 90%. Least satisfaction was reported in relation to the conditions of financing (79%) and duration and timing of the programme (88%). Also at the level of institutional beneficiaries within our case studies, a response of one interviewee was that there is a lot of form-filling, but overall he thought the management requirement was not too bad in the main. The same respondent thought that budget sheets need clarifying and made more flexible, especially for staffing. He commented that it is impossible to anticipate such detail in advance and there needs to be greater flexibility for which category of staff will be deployed on the different tasks as the project progresses. There are unnecessary sub-divisions and it is over-fussy. Another institutional interviewee remarked that the lead time between receiving notice of award and its commencement was very short, allowing little time for planning and getting 'up to speed' at the commencement of the award. In conclusion there is high satisfaction in this area based on the survey, and although the remarks from some individual interviewees indicate some moderately serious administrative issues that they emphasised that they wished attention brought to Adequacy of management resources. Do the stakeholders and beneficiaries possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? At the level of beneficiaries, a response of one interviewee at the LSE was that the management of the resources is very time-consuming and therefore expensive and should have been included in the funding model. Another respondent adds to this by stating that, with regard to funding and expenditure, the guidelines are time-consuming to understand and this can cause delays. For example, she had to work out how they could employ an hourly research assistant because the detailed criteria of JM do not fit with the criteria that they use to employ assistants at the LSE. These criteria do not translate well and it seems that the length of contract that the JM funding allowed was too short. As a consequence, it was not realistic to set up a contract for seven days. Trying to resolve this issue was very time-consuming. Part of the issue is that JM Programme is trying to micro-manage the funding and this is not always realistic and can create a lot of difficulties. The Centre of Excellence also reports that whilst the JM Action funds a number of activities, and though not a huge amount, it helps to subsidise ongoing projects. The status of receiving the money is important since in the UK there are few other public resources for engaging with civil society. The funding allows the beneficiary to spread resources over a number of activities and that flexibility is very attractive. However, the regulatory cost is high so there is little effect that the JM work can have on the institution. These remarks appeared regardless of assurance from the representative of the EC that the JM programme has adopted some efficiencies e.g. streamline accounting from experience of other LLP sub-programmes. The beneficiaries also reported that there is an insufficient budget for dissemination especially for the skills needed to develop electronic means of doing so. This is vital, but it is something that needs to be made easier and quicker since academics are not prepared to spend the time. Our respondent has no confidence that any centrally managed system could facilitate such work, so would want to develop it locally. This remark can be contextualised within a view that centrally organised conferences are very poor, and that claiming expenses is very constraining and time-consuming. An overall view is that the resources are insufficient for the dissemination. Another beneficiary reported that he appreciated the flexibility allowed within the programme - he has been allowed to stretch the budget to include additional relevant events/speakers within his budgetary limit. So here the Jean Monnet Chair at the University of Edinburgh is not reporting the same level of concern as from the LSE, finding the management system and funding approach satisfactory. These contrasting remarks are interesting to reflect upon in the light of an EC perspective from our interviewee that great value is obtained from relatively small grants, and that there is a multiplier effect from networking between projects and reaching out to civil society. 110

111 4.3. A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? Respondent from the EC reported that KA1 and KA3 of the JM Action are monitored through the Executive Agency, who receive project activity reports and gradings. They carry out monitoring visits, provide an overall report and identify problems. The numbers of projects make it difficult to monitor all annually and there is a focus on the largest. Some problems are identified and sanctions are available, but the data available did not identify them specifically. The programme has adopted some efficiencies e.g. streamline accounting from experience of other LLP programmes. At the level of beneficiaries, a response of one interviewee was that the requirement to keep time sheets is a concern. If the intention of such detailed form-filling is to cut fraud it is unlikely to work. Keeping a record of the work carried out and the numbers of hours needed to do is fine but not having to identify the precise number of hours for the precise work, by the hour, for each individual. It is too time-consuming. The respondent agreed that it is right that beneficiaries have to justify the number of hours that they work, but they should not have timesheets for individuals to fill in. Another response was that the reporting is not onerous, but he was not sure how rigorous the process actually is. The annual report covers both activities and impact in so far as indicators of that are available. His view is that at this stage, in many respects, impact judgments rest on feedback received. 111

112 5. Jean Monnet KA2 Operating Grants to Support Specified Institutions 1. INTRODUCTION This EU-level case study focuses on the implementation of the Key Activity 2 (KA2) of the Jean Monnet programme. The overall purpose is to provide evidence relating to the key evaluation issues of relevance, effectiveness and efficiency of the EU support granted to European University Institute (EUI) in Florence (Italy) in Also, this case study particularly contributes to the evaluation of the efficiency of the LLP, since the in-depth analysis of the KA2 was chosen for assessing the cost-effectiveness of using alternative policy mechanisms for reaching similar results of the LLP at a lower cost. According to the Decision establishing the LLP, the KA2 of the Jean Monnet programme offers operating grants aimed at supporting certain operational and administrative costs of six specified institutions pursuing an aim of European interest. The list of these institutions was set out in the Decision establishing the LLP: - the College of Europe (Bruges and Natolin campuses); - the European University Institute (Florence); - the European Institute of Public Administration (Maastricht); - the Academy of European Law (Trier); - the European Agency for Development in Special Needs Education (Middelfart); - the International Centre for European Training (Nice) 56. As it was stated in the Inception Report, the selection of the EUI for the case study was based on the distribution of financial resources between the supported institutions under the KA2. In the period, the EUI received almost 22 million, which amounts to around 42% of total budget allocated for the KA2 of the Jean Monnet programme and around 32% of total Jean Monnet budget (see Table below). These data indicate that the EUI received the biggest share of the total support available for the specified institutions. Moreover, in the period the EUI experienced the biggest increase in the value of annual operating grants in comparison with other supported institutions. If the value of grant for the EUI increased by around 40% in the period (followed by an increase of 37% for the College of Europe), the annual support for other specified institutions rose only by about 8% in the same period. Table Budget allocations for European University Institute in the period ( ) Pre-Jean Monnet* LLP ( ) Total programme Total sub-programme n/a n/a Action 1 KA European University Institute Note: n/a not available. * Community action programme to promote bodies active at European level and support specific activities in the field of education and training ( ). The case study is based on the following sources of information: - Four semi-structured interviews, incl. two with the representatives of the Directorate General for Education and Culture of the European Commission, one with the representative of the Education, Audiovisual and Culture Executive Agency and one with the representative of the EUI; - Desk research of European legislation and Programme documents, the evaluations of previous programmes, monitoring data, Annual Reports of the President of the EUI, EUI reviews, data sent by the beneficiary etc. The interviews and desk research were carried out during the period of April 14 th September RELEVANCE The main objective of this section is to describe the European Added Value of the selected activity. 56 Article 36 of the Decision No. 1720/2006/EC of the European Parliament and of the Council of 15 November 2006 establishing an action programme in the field of lifelong learning. 112

113 2.1. To what extent do the intermediate and specific objectives of the KA2 remain pertinent to the EU policy priorities? By the means of providing annual operating grants to six specified institutions, the KA2 directly contributes to two (out of four) operational objectives of the Jean Monnet programme: c) to support key European Institutions dealing with issues relating to European integration; and d) to support the existence of highquality European institutions and associations. These two operational objectives are closely linked to the specific objectives of Jean Monnet: a) to stipulate teaching, research and reflection activities in the field of European integration studies; and b) to support the existence of an appropriate range of institutions and associations focusing on issues relating to European integration and on education and training in a European perspective. Besides eleven specific objectives at the LLP level, the specific objectives of Jean Monnet clearly contribute to the intermediate objective of the Programme to foster interchange, cooperation and mobility between education and training systems within the EU. Overall, the aforementioned objectives of the KA2 of the Jean Monnet programme remain highly pertinent to the EU policy priorities. There is a clear need for both wider and deeper knowledge with respect to processes happening in the EU as well as their impact on the Member States and individual citizens. One could even argue that after the public rejection of the Constitutional Treaty in France and the Netherlands in 2005 and later the Lisbon Treaty by the Irish citizens in 2008 high pertinence of the KA2 to the EU priorities was only reinforced. High pertinence of the objectives of the KA2 was confirmed by both the previous evaluations as well as data collected under this case study. The goal of the operating grants is to ensure institutional stability and allow long-term planning, so that EU priorities are continuously and comprehensively addressed at the institutions supported. The evaluation of the previous Community action programme to promote bodies active at European level and support activities in the field of education and training 57 found clear links between the results and impact of operating grants awarded to specified institutions and Education and Training 2010 Work Programme 58. All six institutions named in the Decision establishing the LLP were also supported under the previous programme in the period and through their regular provision of teaching and training as well as producing research of a very high quality contributed particularly strongly to the EU strategic objective of increasing the quality and effectiveness of education and training systems in the EU. Also, the previous evaluation found a moderate link between the results and impact of the supported institutions and strategic objective of Education and Training 2010 of opening up education and training systems to the wider world. Moreover, there was a weaker link found between the institutions activities contributing to informing and educating citizens about the EU and relevant EU policy priorities in the field of communication and citizen engagement 59 (see Table below). In 2009, 4 general and 32 departmental events, directly related to EU policies, were organised by the EUI. Since all currently supported institutions were also supported in the previous period, predictably, the KA2 of the Jean Monnet programme of the LLP similarly strongly contributes to a very similar strategic objective of the new Education and Training 2020 framework of improving the quality and efficiency of education and training. For instance, according to the global ranking of political science departments in the period, the EUI was ranked as the 1 st in Europe and the 5 th in the world 60, confirming the highest quality of its activities. On the basis of excellent research results, the EUI Department of Political and Social Sciences was also selected for the ExcellenceRanking 2009 of the German-based Centre for Higher Education Development 61. Similarly, a moderate link could be also found between the KA2 and the new strategic objective of promoting quality, social cohesion and active citizenship. Being an international institution, the EUI especially contributed to promoting intercultural dialogue and democratic values. Quite favourable full-funding opportunities for PhD students could also be seen as an important factor contributing to the EU 57 Decision No. 791/2004/EC of the European Parliament and of the Council of 21 April 2004 establishing a Community action programme to promote bodies active at European level and support specific activities in the field of education and training. Official Journal of the European Union, L 138/31, Report from Education Council to the European Council on the Concrete Future Objectives of Education and Training Systems. Brussels, 14 February Communication from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions: The Commission s contribution to the period of reflection and beyond: Plan-D for Democracy, Dialogue and Debate. COM(2005) 494 final, Brussels, Simon Hix, A Global Ranking of Political Science Departments. Political Studies Review, 2004, Vol. 2, EUI Review, Winter 2009,

114 horizontal policy of equal opportunities, since it diminished the risk of social selectivity of students. In the period, the EUI spent around 9% of its total budget for student grants. Table Contribution of the KA2 of the Jean Monnet programme to relevant EU policy priorities EU Policy Priority Contribution ET 2010: to improve the quality and effectiveness of EU education and training systems +++ ET 2010: to ensure that they are accessible to all + ET 2010: to open up education and training to the wider world ++ ET 2020: to make lifelong learning and mobility a reality ++ ET 2020: to improve the quality and efficiency of education and training +++ ET 2020: to promote equality, social cohesion and active citizenship ++ ET 2020: to enhance creativity and innovation, including entrepreneurship, at all levels of education and training + Plan D: to provide citizens with the information and the tools to actively participate in the [EU] decision making process ++ Plan D: to stimulate a wider debate between the European Union s democratic institutions and citizens ++ Note: +++ very strong contribution; ++ some contribution; + no direct contribution. Source: ECOTEC, Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report., 29 June Contribution of the EUI to the Education and Training 2020 strategic framework (marked in bold) was evaluated by the contractor To what extent do the intermediate and specific objectives of the activity remain pertinent to the needs of the target groups of the KA2? The target audience of the KA2 of the Jean Monnet programme can be divided into two main groups: direct institutional beneficiaries and indirect individual beneficiaries. Direct institutional beneficiaries are six specified institutions named in the Decision establishing the LLP. Indirect individual beneficiaries are students, teachers, researchers and other educational staff within those institutions, policy-makers as well as wider public. Additionally, different research centres and bodies working in the field of European integration could be also added to the list of potential indirect beneficiaries of activities pursued by the institutions supported under the KA2. Article 2 of the Convention setting up the EUI states that the aim of the EUI is to contribute, by its activities in the field of higher education and research, to the development of cultural and scientific heritage of Europe 62. This aim is pursued by the means of teaching and research at the highest level in a European perspective in Economics, Law, History and Civilization, and the Political and Social Sciences. Besides, the EUI seeks to be a forum for exchange and discussion of ideas and experience 63 in the fields of its teaching and research. These aims correspond to the objectives of the KA2 of the Jean Monnet Programme and remain highly pertinent to the needs of both target groups. Concerning the direct institutional beneficiaries of the KA2, the Jean Monnet programme substantially supported the activities of specified institutions, allowing them to pursue their objectives, develop the existing excellence and contribute to filling-up the knowledge gap in the field of European integration. It could also be concluded from the interviews that the support from the Jean Monnet programme substantially contributed to the prestige of these institutions and their ability to develop long-term, sustainable curricula and academic networks. No surveys or interviews with the indirect beneficiaries of the KA2 were carried out for this case study. However, the desk research shows that the objectives of the KA2 remain highly pertinent to the needs of the target groups at least for three main reasons. First, the total number of applications for the PhD programme increased from in 2007 to in 2009, making the competition of nearly eight applicants per grant. Although there were minor fluctuations in the number of received applications for studies in the Department of Economics and the Department of History and Civilization, the total number of applications increased in all four departments of the EUI (see Figure below). A similar trend is observed in the number of applications received for the Max Weber Programme for Post-doctoral Studies the largest post-doctoral programme in the social and human sciences in the world. This programme attracted 446 applicants in 2006, 62 Article 2 of the Convention setting up a European University Institute as revised by the 1992 amending Convention. 63 Ibid. 114

115 784 in 2007, 928 in and as many as in (the number of applications more than doubled). Second, the data indicate that the Institute s graduates were very successful in the labour market. A recent exit survey performed by the EUI showed that around 70% of its researchers finishing their PhD studies found positions in academia, and others outside the academic sector (mainly in international organisations) 66. In view of the latter fact, it is hardly surprising that according to the evaluation of the previous programme, more than two thirds of EUI students thought that attending the Institution enhanced their career prospects. 67 Third, research produced at the EUI as well as its various activities are well recognised by academia and policy-makers, what is confirmed by a high place of the EUI in different international rankings mentioned above. It could be concluded that the good recognition of the EUI substantially contributed to the objective of being a central place of discussion and exchange of ideas and experience on different issues of great importance to the EU. As mentioned above, 36 events were organised in direct relevance to EU policies, and many others included prominent guest-speakers, such as politicians and key academics. Grantees are expected to study European or comparative topics. Figure Number of applications for the PhD programme Number of Applications by Departments of the European University Institute Economics History and Civilization Law Political and Social Sciences Source: The EUI President s Annual Report, Spring 2010, p What is the European Added Value? Would other international or national schemes / instruments provide enough support to the activities funded by the KA2, if there was no LLP? The European valued added of the KA2 is high, since the LLP supports the institutions that are unique and have already proven a record of excellence. The interview programme confirmed that the underlying rationale behind the current system of funding under the KA2 is to build on existing excellence and develop it further. The EUI is a clear example of this. The LLP legal base states that the KA2 of the Jean Monnet programme offers annual grants to support certain operational and administrative costs of six institutions pursuing an aim of European interest. Although the aim of European interest was not defined in the Decision establishing the LLP, the EU supports unique institutions that have already proven a record of excellence and whose objectives and main activities meet the objectives of the Jean Monnet programme and the LLP. As it has been already stated, the EUI objectives clearly correspond to the operational and specific objectives of the Jean Monnet programme as well as specific and intermediate objectives of the LLP. Besides, the data aggregated for this study show that the EUI is a unique and well recognised institution at least for four reasons: 64 Ramon Marimon, Karin Tilmans, Michele Grigolo, Max Weber Programme Self-Evaluation Report , September 2009, p Data provided by EUI secretary general, Mr Marco Del Panta. 66 EUI Review, Winter 2009, p Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report, 29 June 2007, p

116 - the EUI is distinguished by its general reputation in the field of European studies, since the Institution produces research in a European perspective in history, law, economics, political and social sciences of a very high quality; - the EUI offers one of the largest doctoral and post-doctoral programmes in social science in the world and is exclusively dedicated to doctoral and post-doctoral training (is a doctoral school ); - the EUI provides full funding for students, spending around 9% of its total budget of more than 45 million for student grants; - the EUI is a truly international institution, creating a community of over 50 different nationalities working and studying in a one place with no dominant national culture. Since the activities pursued by the EUI are distinguished by their strong European perspective and the EUI is widely recognised as a unique institution in its teaching and research field, the EU support offers much better opportunities for pursuing its objectives than could be offered by other national and international instruments. One of the most important reasons for high European added value comes from certain side effects of awarded grant, namely, the prestige given to the EUI as well as a potential for better links with policy-makers as a result. The latter side effects are related to the introduction of the European dimension into the activities of the EU. This dimension increases visibility of the EUI activities, provides a quality label to the Institution (a combination of academic independence and links to EU institutions), helps to attract larger target groups of the Jean Monnet programme and creates better chances of attracting additional funding (e.g. from national governments, FP7). Moreover, one could find many arguments in favour of concentrating funds on a few specific institutions in order to further develop the existing excellence rather than channelling many smaller grants to a larger number of them (smaller grants are awarded to a much larger number of institutions under the KA1 of the Jean Monnet programme; see the EU level case study on the KA1 of the Jean Monnet programme). It may be difficult to estimate a precise added value of side effects of EU grants such as European dimension introduced into teaching and research, as well as their relationship to the financial inputs. However, our interviewees stress that regular re-evaluation ensures that the selected institution outperforms its alternatives in this respect. The total budget of the EUI increased from over 35 million in 2004 to over 45 million in While the share of support from contracting countries decreased from 56% to 52% in the period (not without the impact of the economic downturn, yet overall, value of support was nevertheless constantly increasing), the share of EU award increased from 13% to 17% in the same period (see Figure below). On the one hand, the EU contributions make a substantial part in the EUI budget. On the other hand, the analysis shows that outputs of the EUI cannot be distinguished by the source of the EUI s revenues. To be more specific, the EUI (similarly to other specified institutions) presents all the outputs and results of its activities in its regular annual reports, but it is not possible to distinguish the precise additional value of the EU grant or the extent to which these outputs and results are being reached because of EU contributions. On the other hand, the operating grant aims at supporting institutional development, sustainability and long-term commitment to the development of European curricula. Meanwhile, EU institutions do not interfere into the academic autonomy of the EUI. Therefore it is expectable that the grant is not directly related to particular outputs. Figure The share of different sources of revenues in the total EUI budget The Share of Different Sources of EUI Revenues 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Source: The EUI President s Annual Reports Miscellaneous revenue Funding of EC Historical Archives Other assigned revenue Proceeds from research services Contributions from European Communities Deductions from staff remunerations Contributions from Contracting states 116

117 2.4. What evidence exists of synergies between the KA2 and other different actions within the subprogrammes? Do duplications, overlaps or other disadvantageous issues exist? Synergies between the KA2, other key activities of the Jean Monnet programme and other sub-programmes are rather limited mainly because of big segmentation of the LLP. If one could find small synergies between sectoral sub-programmes due to a great variety of supported activities, different implementation mechanisms and target groups, the Jean Monnet programme seems to be somewhat even more detached for all the above-mentioned reasons. Moreover, the grants are awarded to the pre-selected institutions. There is no open competition for operating grants and the institutions are able to use the EU contributions according to their own needs. As a result, possible synergies of activities pursued by these institutions largely depend on their initiative. There may be valid reasons to think that more competition for operating grants could lead to better outputs of the KA2 as well as bigger synergies with other sub-programmes (e.g. Erasmus). However, according to our interviewees, opening-up the action to full competition would be neither politically feasible, nor effective, since the long-term planning produces the aforementioned sideeffects and the quality label. Synergies could be sought from within other sub-programmes of the LLP, as the regulations of operational grant only influence long-term goals rather than short-term academic decisions of the EUI. The fact that so many EUI graduates become academics could also be exploited in building academic networks and looking for mobility and exchange synergies within the LLP. Despite rather limited synergies, the analysis found no evidence of duplication. First, the Jean Monnet programme aims at supporting teaching and training in a European perspective and in this regard differentiates itself from other sub-programmes. Second, the KA2 is a unique action category, since it supports certain institutions rather than specific activities. There are no similar mechanisms to operating grants used for implementing the LLP under other sub-programmes. Third, the direct target group of the KA2 is the specific institutions with an international reputation for excellence. Moreover, in terms of indirect beneficiaries, KA2 supports the highest academic level - PhD and post-doctoral studies. Therefore, uniqueness of the KA2 could be considered to be a pre-condition for preventing duplication. However, the biggest risk of overlaps or even duplications comes from concrete activities pursued by the selected institutions themselves. If the institutions become more similar in terms of their activities, the overlaps will undoubtedly appear. The tendency of the selected institutions to move closer together < > in terms of their activities was noticed in the external evaluation of the previous programme 68. However, in the period the EUI could be clearly considered as a unique institution at least because of its concentration on doctoral and post-doctoral training the activities that were not pursued by other pre-selected institutions. 3. EFFECTIVENESS The main objective of this chapter is to provide assessment of the attainment of goals of the selected institution and proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the activity could be achieved Effectiveness in achieving targets and contributing to priorities What is the level of the Programme implementation at the EU level? What is the extent to which the beneficiaries are satisfied with the outputs and results of the KA2? The evaluation of the major outputs, results and outcomes of the EUI shows that overall the EUI highly contributes to specific and intermediate objectives of both the Jean Monnet programme and the LLP. The data collected for this case study also indicate that the beneficiaries are quite satisfied with the outputs and results of the EUI. As a result, it can be concluded from the analysis that the KA2 is implemented at the EU level to a high extent. However, it is important to reiterate the previously mentioned finding that it is difficult to assess a precise added value of the EU contributions to the EUI (inputs), since the outputs, results and impacts of the EUI activities are not separated according to different sources of revenues received by the Institution (see Figure 3.5.5). In the period, the EUI officially consisted of four academic departments, a Centre for Advanced studies and was home to the Max Weber post-doctoral programme, which used to be financed separately 68 Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report, 29 June 2007, p

118 and was included as a separate budget line, but two years ago has become an integral part of the operating grant. In 2009, these departments, the centre and the programme hosted a scientific community of around members: the EU had 55 full-time faculty, 550 doctoral candidates, 100 post-doctoral fellows, 100 visiting faculty and fellows and 150 administrative staff 69. According to the analysis summarised in Table below, the EUI pursued three main groups of activities: (1) provided teaching and training; (2) produced high quality research; and (3) maintained a forum for discussion and exchange of knowledge and ideas. It can be concluded that these outputs directly contributed to such results as better skills and knowledge of EUI students and researchers, greater overall knowledge in the field of European integration and better links between the EUI and other European as well as non- European higher education institutions and public authorities. In addition, according to an EUI representative, there is a spillover effect, since graduates often become academics in other institutions. If the first two results were directly related to the outcomes of the LLM programme, the doctoral programme, the post-doctoral programmes and high quality research respectively, the third result was to a considerable degree conditioned by different events organised by the EUI (see the previous section). These events offered a place for students, academics, public officials and wider public to engage in discussions on topics relevant to the EU and the European society and contributed to the objective of the EUI to be a forum for exchange and discussion of ideas and experience. Finally, the outputs and results of the activities pursued by the EUI were transformed into increased employability of EUI students and researchers, greater knowledge in social and human sciences, better informed policy-making and increased interchange, cooperation and mobility between education and training systems within and outside the EU (impacts). It is important to note that the last-mentioned outcome of the EUI is almost the same as the intermediate objective of the LLP. Nevertheless, this does not mean that only different events organised by the EUI contributed to the implementation of the intermediate objectives of the Programme, since all the major groups of outputs of the EUI (teaching/training, research and discussions) were closely interrelated and at the same time inseparable of each other. Table Outputs, results and impacts of the activities pursued by the European University Institute in the period Outputs Results Impacts Four-year doctoral programme in Economics, History and Civilization, Law, Political and Social Sciences One-year Master of Laws (LL.M.) programme in Comparative, European and International Law Various post-doctoral programmes, incl. Max Weber Fellowships, Jean Monnet Fellowships, Bernard Braudel Fellowships and other High quality research in a European perspective in the fields covered by the Departments of Economics, History and Civilization, Law, Political and Social Sciences and Robert Schuman Centre for Advanced studies Large number of different events (14 general and numerous departmental events in 2009), incl. seminars, workshops, conferences and summer schools, contributing to the objective of being a forum for exchange and discussion of ideas and experience Better skills and knowledge of students and researchers Greater knowledge in the field of European integration and society Better links between the EUI, other higher education institutions and public authorities Increased employability of students and researchers Greater knowledge in the field of social and human sciences Better informed policy-making and increased interchange, cooperation and mobility between education and training systems within and outside the EU The high level of implementation of the KA2 as well as satisfaction of the beneficiaries is confirmed by a number of facts, some of which have been already mentioned. First, available data show that in the period the EUI managed to attract a growing number of applicants for both of its two major teaching programmes: the doctoral programme and the Max Weber post-doctoral programme. Although the 69 EUI Brochure: A Community of scholars

119 contractor did not carry out a survey of the beneficiaries of the KA2 of the Jean Monnet programme, a growing number of applications could be used as a good proxy showing a quite high satisfaction of the target groups. A very high overall completion rate of the doctoral programme (85% 70 ) also indicates high results of the programme. At the same time new data reinforces the finding of the evaluation of the previous programme that the beneficiaries of the EUI very successfully integrated into the labour market (mainly in academia and international organisations) 71 (see question 2.2). Regarding the research activities of the EUI, it is rather difficult to evaluate the impact of the EUI research products on the fields of European integration studies or wider social and human sciences. However, the internal evaluation by the staff of the EUI shows that the impact of Institution research products has been increasing 72. This is mainly related to the fact that in the period the Institute managed to increase its proceeds from research services from 6,1 million in 2006 to 7,2 million in 2009 (over 19%) 73. In the same period the EUI produced around 2500 books, contributions to books, articles in journals and working papers. On the one hand, it should be noted that the EUI research output decreased from over 900 products in 2007 to around 800 in 2008 and less than 800 in On the other hand, it is important to highlight that pure statistical facts do not allow making any definite conclusions about the quality and impact of the research output What are the main external factors affecting the outputs and results of the KA2? One of the main external factors, which is directly related to attracting applicants from all EU Member States, was the availability of grants to doctoral students. Since most of grants are awarded by national grant-awarding authorities, the availability of them for certain nationals depends on specific countries, which are contracting members of the EUI. Of the MSs grants allocated, 118 were given to applicants from EU15 countries, and only 15 to EU12 applicants, signalling significant disproportion. In the period the EUI convention was not signed by six new EU Member States: Bulgaria, the Czech Republic, Hungary, Lithuania, Malta, and Slovakia. As a result, there were no grants available for PhD students from these countries, but they were exempted from paying tuition fees. It is anticipated that most of the remaining Member States will most probably join the EUI in the near future. 75 However, some of the new Member States were constrained by their critical financial situation and the current economic downturn. A discussion has been started whether the grants should be unlinked from the applicants nationality. Another external factor is related to the availability of external funds for research. Despite a consistent growth in the value of externally funded research activities, the biggest part of EUI resources came from the EU (including FP7). This situation was mainly linked to the fact that the international status of the EUI did not permit the Institute to access national research foundations. One of the solutions could be a creation of a European-type of foundation 76, since a limited access to national research foundations clearly reflects too narrow boundaries of the current European Research Area. However, due to the EU legal-base, such EUlevel developments depend to a large degree on the willingness of the Member States Effectiveness in implementing horizontal policies: How does the implementation of the KA2 contribute to the horizontal priority of multilingualism? The KA2 of the Jean Monnet programme does not directly contribute to the implementation of multilingualism. Nevertheless, it can be concluded that the contribution of the KA2 to the priority depends on the multilingualism policy implemented by the supported institutions themselves. Concerning the EUI, the analysis shows that the EUI highly contributed to the EU horizontal policy of multilingualism. First, the EUI is a unique institution in view of having nine official languages set out in the Convention setting up the EUI. These languages include Danish, Dutch, English, French, German, Greek, Italian, Portuguese and 70 EUI Review, Winter 2009, p Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report, 29 June 2007, p The EUI President s Annual Report, Spring 2010, p In the period of the EUI revenues from research activities increased by around 66%. Source: The EUI President s Annual Reports The EUI President s Annual Report, Spring 2010, p Ibid., p Ibid., p

120 Spanish. Second, according to the Convention, there is an obligation to use two working languages for each academic activity. Third, all the EUI teaching staff and research students should be able to adequately use two official languages mentioned above. 77 For example, it is interesting to note that in practice every applicant for the doctoral programme is able to choose an application form in one of five languages, namely, English, French, German, Italian or Spanish. This policy pursued by the EUI largely corresponds to the EU policy of multilingualism. However, two considerations could be taken into account. First, the requirement for the EUI teaching staff and particularly research students to have adequate knowledge of at least two official languages of the EUI means less than the EU priority of mastering two foreign languages 78, since a student s national language may coincide with one of official languages of the EU. Second, no EU12 language is included (e.g. Polish has 40 million speakers, Danish - 6 million), and the requirement of using two languages from those listed above may constitute a disadvantage to EU12 students, already disadvantaged due to a limited number of grants How does the implementation of the KA2 contribute to the horizontal priority of equal opportunities? Similarly to the EU multilingualism policy, the institution selected for this case study does not directly contribute to the implementation of the EU horizontal policy of equal opportunities. The EC fully trusts the institutions to ensure that equal opportunities are respected at all levels. As a result, the indirect contribution also depends on concrete policy measures pursued by the supported institutions themselves. It can be concluded from the evaluation of the support awarded to the EUI that the Institution contributed to equal opportunities to a rather high extent. In this respect, the KA2 contributed to equal opportunities indirectly to a rather high extent, too. First, the EUI made noticeable effort to ensure gender balance at the EUI. As it can be seen from Table below, in the period the gender balance in the doctoral programme had been constantly improving. The gender balance was quite good in the Max Weber post-doctoral programme too, with the share of selected female applicants in 2009 being even bigger than that of male applicants. However, the available data indicate that male candidates are usually more active not only in applying for post-doctoral fellowships (the applications received for doctoral programme was almost balanced), but also for professorial positions, which are not shown in Table below. For instance, although the EUI received 1341 applications for vacant professorial positions in the period, only 259 (19%) of all applications were from female candidates. In line with the percentage of received applications, only 13 (24%) women were selected and only 9 (17%) decided to take up their appointment. As a result, in 2009 only 19% of all EUI professors were female. Nevertheless, it is important to note that the percentage of women in professorial positions in 2002 was 15%. The desk research also shows an improvement in the gender balance of selection committees for vacant positions and especially senior administrative positions, with the share of female employees being around 44% in 2009 (a sharp increase of about 30%, when comparing with 2002) 79. Table Number of applicants and registered students by gender Female Male Female Male Female Male Number of registered researchers to the doctoral programme 41,2% 58,8% 44,7% 55,3% 49,3% 50,7% Number of applications received to the n/a n/a Max Weber post-doctoral programme 39,2% 60,8% 43,0% 57,0% n/a n/a Number of registered researchers to 2007/ / /2010 the Max Weber post-doctoral programme 47,1% 52,9% 46,3% 53,7% 54,8% 45,2% Note: n/a not available. Source: The EUI President s Annual Report, Spring 2010, p. 26; Ramon Marimon, Karin Tilmans, Michele Grigolo, Max Weber Programme Self-Evaluation Report , September 2009, p. 31, Article 27 of the Convention setting up a European University Institute as revised by the 1992 amending Convention. 78 E.g., Council Resolution of 21 November 2008 on a European strategy for multilingualism. Official Journal C 320, ; Council conclusions of 12 May 2009 on a strategic framework for European cooperation in education and training ( ET 2020 ), Official Journal C119/2, The EUI President s Annual Report, Spring 2010, p

121 Second, full-funding provided for the majority of selected students was an essential prerequisite for decreasing entrance barriers faced by socially or/and economically disadvantaged students/researchers. For example, according to recent statistics, around 160 research grants were awarded to PhD students each year. These grants were usually awarded by Member States and were available for nationals from 21 EU Member States and 2 Associate Member States. The Italian Ministry of Foreign Affairs and the Spanish Ministry of Foreign Affairs also awarded grants for candidates from certain non-eu countries, incl. Turkey, Russia and the former Soviet countries, countries in Latin America, Middle East, Northern Africa, etc. However, it is important to highlight that grants were not available for all the nationals from all EU12 members, but they were exempted from tuition fees Effectiveness in integrating previous activities into the LLP How successfully has the integration of the previous activities into the LLP been implemented? What are the advantages and disadvantages of the integration of the previous programmes into the LLP? Has the integration of the previous activities influenced the target groups of the KA2? The previous Community action programme to promote bodies active at European level and support activities in the field of education and training supported seven specified institutions under its Action 1: six specified institutions currently supported under the KA2 of the Jean Monnet programme of the LLP and European Inter-University Centre for Human Rights and Democratisation (EIUC). Since the EIUC is currently supported under the financing instrument for the promotion of democracy and human rights worldwide 80, the integration of the previous activities had no influence on the number of direct institutional beneficiaries. Analysis shows that after the launch of the LLP, the priorities of the Jean Monnet programme remained unchanged. Moreover, it could be concluded from the longitudinal mapping of the LLP that the integration also did not affect the number of activities supported under the previous programme. However, all six institutions experienced a slight increase in the value of awarded operating grants, with the biggest increase (around 24% in 2007, when comparing with 2006) experienced by the EUI (see Table above). Also, several interviewees noted that the integration of the previous programmes boosted the status of the Jean Monnet programme and increased its visibility. As a result, it can be concluded that, on the one hand, the integration resulted in bigger prestige given to supported institutions, visibility and political attention, which helps to attract larger target groups, ensure better links with policy-makers, attract additional funding, etc. On the other hand, such effects of integration are difficult to measure, as year-toyear continuity and stability rather than LLP integration were identified as the key factors of institutional prestige Effectiveness in reaching the target groups and disseminating results How successful is the LLP in reaching the target groups of the KA2? Overall, the target groups of the EUI are reached quite successfully. This finding is based on a growing number of applications received for both doctoral and post-doctoral programmes offered by the EUI (see section 2.2). However, it should be highlighted that the demand for places at post-doctoral programmes is much higher than the EUI currently offers. For instance, although in the period the success rate of applications for the doctoral programme was 12-13%, the success rate of applications for the Max Webber post-doctoral fellowships was only 5,2% in 2007 and even decreased to 4,5% in On the one hand, these data indicate good visibility of the EUI amongst recent PhD graduates and good general reputation of the EUI both inside and outside the EU. High competition should be also seen as a big advantage, since it allows the Institution to select the best candidates. On the other hand, such low success rates of applications for the post-doctoral programmes show that the current supply does not meet the demand and many wellqualified candidates become inevitably disappointed, especially as competition is different in the MSs (e.g. over 16 applicants per PhD grant in Italy, but only 3 applicants/grant in Switzerland). Besides low success rates of applications for the post-doctoral programmes, one must pay attention to certain target groups, which are currently underrepresented in the EUI. First, the analysis shows underrepresentation of candidates from certain Member States, especially EU12, what was also highlighted 80 Regulation (EC) No. 1889/2006 of the European Parliament and of the Council of 20 December 2006 on establishing a financing instrument for the promotion of democracy and human rights worldwide. Official Journal L386/1,

122 in the evaluation of the previous programme 81 and is reflected in the data provided by our informant at EUI. Concerning the doctoral programme, it is important to iterate the fact that the EU enlargement was in variance with the number of new contracting countries of the EU. At present, the EUI convention has not been signed by six EU12 Member States: Bulgaria, the Czech Republic, Hungary, Lithuania, Malta, and Slovakia. As a result, not all the EU nationals were eligible for the EUI grant, which could be seen as a barrier to continuing studies for potential candidates from these countries. Application figures allow concluding that the Max Webber programme was much less successful in attracting candidates from the Baltic States and the Scandinavian countries 82. Second, while gender balance was not a significant issue amongst the applicants for different programmes, the former President of the EUI Yves Meny highlighted in its latest annual report that the EUI did not sufficiently represent all of Europe s ethnic diversity, notably, children of foreign migrants. Moreover, although more than 50 nationalities were represented in the EUI, the President encouraged to make more effort to increase the diversity and pluralism of the EUI further and to attract more candidates from non- European countries 83. One explanation about the difficulties in attracting non-european applicants would be that the programmes offered by the EUI are more visible in Europe than in other parts of the world. In relation to the doctoral programme, one could also argue that candidates from non-european countries have less funding opportunities than the Europeans. In 2009, 20 out of EUI s 159 PhD grants were available to non-eea citizens To what extent are the KA2 results adequately disseminated and exploited at both horizontal and vertical levels? As it was already stated, the KA2 of the Jean Monnet programme is aimed at supporting certain unique institutions contributing to the European integration process. Since the EU grant is awarded to institutions, which have a lasting international reputation for excellence, the operating grant (as well as recognition by the EU) is supposed to facilitate the process of developing the existing excellence further. As a result, the initial and long-term impacts (respectively the results and outcomes) of the KA2 are first and foremost the results and outcomes of the supported institutions. The desk research shows that the supported institutions disseminated the results and outcomes of their activities (or the results and outcomes of the KA2) mainly themselves and often informally. Concerning the EUI, it can be concluded that the dissemination was done to a high level. To begin with, the reviews of all the yearly achievements of the Institution could be found in the President s Annual Reports. The EUI also had its magazine the EUI Review, which was published three times a year. Each of its issue was dedicated to discuss a certain topic and, together with the President s Annual Report, could be easily found in the Institution s official webpage. Second, every interested person was provided with access to the database of all the publications produced by the EUI community, including articles, books, contributions to books, doctoral theses, research project reports, working papers and even distinguished lectures. 84 Third, perhaps the most influential mean of dissemination in terms of potential influence to public policy was numerous public events organised by its Schuman Centre for Advanced Studies and to a lesser extent four departments. These events were aimed at bringing together researchers, experts and public-policy makers in pursuance of discussing different issues important for the EU integration. The EUI collects and publishes all press coverage of these events. In 2009, 100 articles appeared in the press of various countries, mentioning the EUI. Special events and research activities or conferences also received substantial attention. 85 Moreover, most of the EUI staff were also regularly involved in the work of European, international and national public institutions and private organisations as consultants, advisors, co-organisers of different discussion forums, interviewees to television and radio, etc. 86 Finally, it is important to highlight the finding of the evaluation of the previous programme, which was also repeated in the latest European Commission s Activity Report , that the training activities organised by the supported specified institutions had 81 Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report, 29 June 2007, p Ramon Marimon, Karin Tilmans, Michele Grigolo, Max Weber Programme Self-Evaluation Report , September 2009, p The EUI President s Annual Report, Spring 2010, p All the publications are accessed through CADMUS (accessed through the official EUI webpage), which is defined as a comprehensive source of bibliographical data of publications by the EUI community. 85 European University Institute, Communications and public relations unit Press review. Volume I: The European University Institute See, for example, Brochure of the Robert Schuman Centre for Advanced Studies, p

123 positive effect on the individuals and authorities involved in the application and implementation of EU law and policy 87. In 2011 a publication is expected, summarising the material gathered through the seminars that took place at EUI. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of the KA2 management and control system sufficient? To what extent is the implementation and management structure of the KA2 efficient and well functioning? According to the Convention setting up the EUI, the Institute has three authorities with a clear division of responsibilities: the High Council composed of representatives of the Governments of the Contracting States; the Principle of the Institute appointed by the High Council; and the Academic Council composed of members of the EUI community and their representatives. The High Council is the highest authority directing and supervising the activities of the Institute and its meetings are open to the representatives of the European Commission. According to our interviewees, close contact is kept with the EC, but the EC does not want to be an institution which both gives funds and participates in governance. Therefore the influence of an EC representative in the High Council is limited. The High Council approves the Institute s annual budget. The Principle implements the budget and reports to the Council in respect of the implementation. In the process of monitoring the implementation the budget, the High Council is assisted by two auditors, who regularly prepare their annual reports. Moreover, the Institute has its own Internal Audit Service, whose purpose is to contribute to the quality of management and control systems as well as sound financial management of the EUI funds. A challenge is often finding external auditors with good knowledge of EU regulations and the particular situation of the EUI. Regarding the activities of the Institute, the President of the EUI regularly prepares his annual reports. They are examined by the Academic Council and submitted to the High Council. It is also important to note that besides annual reports of the President about the Institute s activities and reports prepared by the auditors about the financial issues of the Institute, the President also presented a strategic review of his eight-year presidential term in the period. Meanwhile, the departments prepared regular self-evaluation reports (incl. the self-evaluation report of the Max Weber programme ). The Institute also conducted various surveys about the carrier of their alumni, satisfaction of participants of different activities, etc. Concerning relations with the European Commission, one must pay attention to several problems noticed by the external evaluation of the previous programme, namely, a lack of technical coherence across DGs ; constant changes in Financial Regulations ; too big administrative burden of financial auditing procedures; and insufficient flexibility of the operating grant system. Regarding the final aspect, the former evaluation found out that the ear-marked contribution system had been considered more flexible than the operating grant system. 88 Our interviewees suggest that the main issues have been addressed in the current programme. They admit that administrative burden is high, yet, according to the DG EAC representative, it is inevitable when public money is spent Could the use of other policy mechanisms have provided better cost-effectiveness? The activities of the LLP can be implemented at least by seven different implementation mechanisms: (1) calls for proposals for projects; (2) calls for proposals for operating grants; (3) direct provision of operating grants; (4) calls for tenders; (5) calls for experts; (6) framework contracts; and (7) directly by the European Commission, which may organise diverse actions aimed at supporting the implementation of the LLP. As it was stated above, in the period the EUI as well as other five specified institutions were awarded 87 DG Education and Culture, Lifelong Learning Programme : Activity Report for 2007 and 2008, 2010, p Final evaluation of the Community s action programme to promote bodies active at European level and support specific activities in the field of Education and Training. Final Report, 29 June 2007, p

124 operating grants to cover certain operational and administrative costs of their activities. In other words, the European Commission supported six pre-selected institutions using the mechanism of direct financial support under the KA2 of the Jean Monnet programme. The analysis of seven implementation mechanisms indicates that theoretically only two alternative policy mechanisms could be used for reaching the objectives of the KA2 of the Jean Monnet programme and Jean Monnet programme as a whole: calls for proposals for projects and calls for proposals for operating grants. Meanwhile, all the other implementation mechanisms are rather insufficient to achieve the goals of KA2 or simply not feasible in view of the objectives of Programme. For instance, calls for tenders, experts and framework contracts are rather bounded to buying specific and well-defined services contributing to the implementation of the LLP (e.g., evaluating received proposals, providing consultancy services, etc.). At the same time, regular activities pursued by such institutions as the EUI fall outside the scope of the European Commission s (or the Education, Audiovisual and Culture Executive Agency s) competence. Therefore, it could be hypothesised that using one of two alternative mechanisms, namely, calls for proposals or operating grants, could have provided better cost-effectiveness of the KA2. However, the comparison of direct financial support and alternative implementation mechanisms shows that other mechanisms would not have provided better cost-effectiveness (see Table below). The main advantages of the currently used mechanism come from the concentration of limited financial resources on several institutions with a proved record of excellence, with full respect to their academic independence. Since operating grants are awarded for the whole Programme period, the pre-selected institutions have a constant source of revenues, allowing them to advance their excellence. Such EU recognition also contributes to bigger visibility of the supported institutions as well as gives benefits from bigger political attention and support. Moreover, one should not forget the principle of subsidiarity that can be adapted to this context too. It can be argued that the EU support in the form of operating grants is much more effective than support for certain projects, since the institutions and their community can know the most effective and efficient ways of reaching their objectives much better than supporters. Meanwhile, annual reporting and close contact ensures that the EC is kept updated about the progress of the strategic objectives. E.g. financial contributions can only be raised if there is solid evidence of how extra funding will contribute to better progress towards KA2 objectives. One can also distinguish several disadvantages of the currently used mechanism of operating grants, namely, the lack of competition and unclear links between the EU grants from one side and outputs of the supported institutions from another. Although the some interviewees endorsed the fact that opening-up the action category of operating grants to bigger competition could potentially lead to better results, it was also found out that such changes are not politically feasible because of considerable political support to the preselected institutions, as well as ineffective having in mind the nature of the research carried out. Also, the analysis indicates that it is rather difficult to evaluate to what extent and which outputs/results were achieved precisely because of EU contributions. For example, the operating and administrative costs of the EUI were covered by several sources of revenues (the EU operating grant constituted only around 17% of its total budget in 2009; see Figure above). Regarding calls for proposals for projects, this alternative mechanism would have the advantage of a clear link between the EU support and concrete outputs/results of supported projects. Meanwhile, similarly to the currently used mechanism, the provision of operating grants to successful institutions that win open competition would allow them to be more flexible and freely plan activities in the most effective and efficient way, i.e. without any constraints of allocating the received funds only to certain projects. However, the analysis shows that both of alternative instruments have more disadvantages than advantages. Calls for proposals for projects would probably lead to a dispersion of limited financial resources for a bigger number of institutions. This could also lead to smaller awards than using the mechanism of operating grants and decrease the certainty upon which successful dissemination among the target groups rests. The support which depends on winning open competition would also make it difficult for institutions to plan their activities. Even more importantly, the action category of calls for proposals already exists and would duplicate the KA1 of the Jean Monnet programme. Concerning calls for proposals for operating grants, this mechanism would also create difficulties for planning activities because of unknown results of competition. Even if operating grants were awarded for a longer period, this would not overcome the disadvantages of obscure link between the EUI inputs and outputs of supported institutions. Finally, in both of alternative cases, open competition would not allow the institutions to benefit from political attention and recognition 124

125 at the same level as today. It is important to know hat according to the interviews, the latter advantage could be sometimes even more important than the value of received grants. Table Comparison of different implementation mechanisms of the Lifelong Learning Programme Implementation mechanism Advantages Disadvantages Direct support (awarding of operating grants) by the European Commission to pre-selected institutions - Financial support for pre-selected institutions with a proven record of excellence; - Concentration of limited financial resources; - Constant source of revenues for pre-selected institutions; - Bigger political attention and promotion; - Benefits of decentralisation (the institutions knows how to reach their objectives the best) - No competition between different supported institutions and other potential beneficiaries (pre-selection); - No clear link between the EU inputs and outputs of the supported institutions Call for proposals for projects Call for proposals for operating grants - Competition between different institutions; - Clearly seen outputs and results of EU contributions - Competition between different institutions; - Benefits of decentralisation (the institutions knows how to reach their objectives the best) - Duplications between the KA1 and the KA2 of the Jean Monnet programme; - Dispersion of limited financial resources; - The budget (and the implementation of the work plan as a result) depends on winning competition - No clear link between the EU inputs and outputs of supported institutions; - The budget (and the implementation of the work plan as a result) depends on winning competition Overall, there are more advantages than disadvantages of using the current implementation mechanism of direct financial support. However, it could be recommended to make the link between the operating grants awarded to the pre-selected institutions and outputs/results of their activities clearer. This could be achieved with changes introduced to the contracting and reporting requirements. Speaking generally, cost-effectiveness of any intervention can be improved by either reducing the costs or increasing the effects (providing bigger funding). The potential of unimproved cost-effectiveness of the KA2 could be assessed on the basis of clear and measurable effects of the EU intervention. However, despite considerable achievements of the supported institutions, such effects are currently difficult to assess. 125

126 6. Erasmus Academic Networks 1. INTRODUCTION An overall purpose of this case study is to provide an in depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the Erasmus Academic Networks action. In the Inception Report, it was noted that conducting a case study on the academic networks in smaller countries eligible for support through the actions concerned might bring added value. For example, Portugal has received support in 2008 for a long-lasting academic network in food studies (ISEKI FOOD 3 - Innovative developments and sustainability). The above-mentioned academic networks project was selected for this case study analysis. The information presented in this case study is based on the following sources: - In-depth interviews to ensure triangulation of viewpoints: a) with the EU-level authorities: the representatives of the Directorate General of Education and Culture, and the Education, Audiovisual and Culture Executive Agency; b) with the institutional beneficiaries of the Erasmus Academic Networks action; and c) with the national authorities in the country where the project coordinator is based (Portugal). In total 13 persons have been interviewed at 5 interviews (the full list of interviews is presented in the Annexes). - Desk research, which sought to analyse action s implementation statistics and other documentation, which was publicly available or provided by the EU level authorities and the institutional beneficiaries. The interviews, desk research and analysis of data were carried out during the period of May 5 th - June 10 th RELEVANCE The main objective of this section is to describe the European Added Value of the selected sub-programme / action To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the European and national policy priorities? Erasmus Academic Networks (EAN) are designed to promote European cooperation and innovation in specific subject areas. They contribute to enhancing quality of teaching in higher education, defining and developing a European dimension within a given academic discipline, furthering innovation and exchanging methodologies and good practices. This is achieved by means of cooperation within the network between higher education institutions, university faculties and departments and may also involve professional associations and enterprises as well as other associations. All networks should bring together an appropriate range of relevant stakeholders concerned by the theme addressed. Cooperation within networks is expected to lead to outcomes which will have a lasting and widespread impact on higher education institutions and their environment across Europe in the field concerned. 89 These objectives of EAN action remain highly pertinent to the European higher education policy objectives. EAN is one of the main instruments ensuring LLP contribution to the European Higher Education Area (EHEA), and is perceived as very beneficial in terms of contributing to the aims of the Bologna process. The idea is that networks should work on the common outputs such as common curricular or quality assurance issues, learn in this process, simultaneously educate all partners in the network, and afterwards try to disseminate good practice at European and national levels. For achieving the objective, the requirement to involve at least one partner from each of all the LLP eligible 31 countries was established. In such way it was ensured that smaller countries or countries less aware of EHEA related issues are invited to learn in the process. In the beginning of the LLP implementation this multi-partner requirement was a stumbling block. Now the authorities are discussing the option of reducing this requirement. 89 LLP Erasmus Academic Networks Fiche N 23: ERA-Net - p

127 By addressing (and predicting) key issues in education and training in a sectoral as well as cross-sectoral context, the networks are expected to ensure mutually productive interaction between policy and programme, at European and national levels. Networks are seen as broader frameworks constituting a European meeting place for the specific education and training community to which it relates. The workgroup on the future programme 90 indicated that it should be possible to define such communities in different ways to meet the needs of the target groups concerned, for example on the basis of subjects, roles and functions (e.g. Network of career counsellors), or institutional types. The objectives of the EAN are also considered as highly pertinent to the higher education policy priorities at the national level. 91 The participation of the national institutions in the European networking activities contributes to fostering the internationalisation of higher education institutions. On the other hand, representatives of the national authorities interviewed expressed concern about lack of access to the programming (e.g. a role in setting the themes and priorities) and implementation of the centralised actions of the LLP (e.g. receiving the data of submitted/funded applications and results produced). Lack of coordination based on dialogue between the national authorities, the national agencies and the EU-level institutions is considered as an obstacle to ensuring effective cooperation and quality of the programme To what extent the intermediate and specific objectives of the EAN remain pertinent to the lifelong learning needs of the target groups of the programme? The pertinence of the objectives to the needs of the target groups is high, and, according to the interviewees, the relevance is increasing. Organisations interviewed for the purposes of this case study noted that the main stimulus for the higher education institutions and individual researchers to participate in the networking activity comes from the access to international visibility ( opening to the world ) and the newest academic knowledge in the field. Other benefits that the participants (coordinators and partners) get from the EAN include: - Broadening and deepening the range and content of study programmes, courses or teaching modules (as an outcome of the network activity); - Improving performance of higher education institutions through specific strategies or measures/actions; - Fostering practices of quality assurance; - Improvement of international profile; - Creating links between higher education institutions and enterprises, private sector; - Greater awareness of European dimension of higher education. The quality of the potential networking results is however discouraged by the lack of substantial funding from the programme. Typically European funding only covers the travel expenses, publications and similar costs, while the work of people involved in the project is an own contribution. 92 This is due to the specificities of the current financial model: the EAN funds up to 60% of the total budget, but the upper limit of funding is 200,000 per year. According to the interviewees, this model inhibits funding of large networks. On the other hand, networks usually result in spin-off multilateral projects funded by the LLP or FP7, or other sources that are used for funding new study programmes, research results, databases, publications etc. From this point of view, EAN are seen as first steps in building a project pipeline that leads to much greater benefits and results for institutions and individuals involved What is the European added value? Would other national schemes / instruments provide enough support to the activities funded by the EAN, if there was no LLP? What would be the likely level of outputs produced? The European added value of the Erasmus Academic Networks is high and is mostly related to the contribution to the EHEA by connecting academic community for developing common outputs that would have an impact on institutional reform. 90 Proposals for future programme generation, Working Group B, Partnerships in Education and Training (2010). 91 In this case the finding is based on evidence collected in Portugal alone. 92 In case of FOOD ISEKI 3, only 33% of the total budget is funded from the EAN. 127

128 In the absence of the LLP EAN the same level of networking activities at the European or even global levels in the higher education sector could not be reached, because there are no similar high-level funding schemes in the higher education sector. The Framework Programme only funds networking in the research sector; and other available schemes are substantially smaller or relate to much smaller projects in scope. The EAN is the only scheme that provides an umbrella support for such a wide range of topics What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist? Several synergies as noted at the interviews have been identified as a result of the LLP integration. So far, one of the main effects of integration on the management of LLP is harmonisation of administrative forms of different actions/sub-programmes that should lead to more efficient management of the Programme. It is hard to evaluate what was the benefit of this standardisation for the applicants of the EAN as the synergies are perceived mostly from the European point of view. For example, in case of the Erasmus networks it is already clear that standardisation has lead to some negative effects. Networks are comprised of many partners and standardised forms filled for all partners constitute up to hundreds of pages. Now the discussion has been started on how to develop a special simplified version for networks projects. Overall, there is a trend towards reducing the barriers between the programmes that may lead to unified management of LLP actions not according the target groups under Leonardo, Comenius, Erasmus and Grundtvig, but according the types of actions (networks, partnerships, mobility). This trend has not fully revealed itself yet, but a possibility to use common rules / administrative forms for different types of actions in the post-2013 period was already discussed during the LLP annual conference in Barcelona. Redesigning the LLP according to the synergies between similar actions in different sub-programmes could also lead to reduction of the number of smaller actions that create less added value and to the creation of the critical mass. However, the main drawbacks of this decision, according to the interviewees, would be related to one sector dominating the support (most probably, the university sector) and loss of ownership of the subprogrammes at the coordinating institutions. What regards duplications, it was found that there is certain overlap between the Erasmus multilateral projects and the Erasmus networks projects. This finding is supported by the interviewed institutional beneficiaries according to them, some activities of the networks projects are funded through EAN, while other networks fund similar activities through multilateral projects. There should be a clear distinction between the award criteria or at least a much clearer distinction in the way these criteria are defined in detail for Networks and Projects respectively. The overlap is especially true in case of the Erasmus Structural Networks. The usefulness of distinction between academic and structural networks has been questioned by the expert group on future programme 93 according to the Group, the distinction has blurred. Hence, according to the interviewee at the Commission, there are plans to eliminate centralised action of the Erasmus Structural Networks from the structure of the LLP starting from the next call for proposals (in 2011). 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the programme action could be achieved Effectiveness in achieving targets and contributing to priorities: What is the progress in achieving action / sub-programme targets against financial progress of the programme? What is the extent to which the beneficiaries are satisfied with the outputs and results of the EAN action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? 93 Proposals for future programme generation, Working Group B, Partnerships in Education and Training (2010). 128

129 Overall, the effectiveness of the EAN action in reaching the targets and objectives can be regarded as satisfactory. Despite relatively good results, the effectiveness of the action could be improved. Average success rate for all the Erasmus centralised sub-actions over the period was about 37% and ranged from 9% to 100% by country. The average success rate of the Erasmus Networks was about 45% (i.e. was highest among other centralised sub-actions). Numbers of submitted applications have been increasing each year, but the share of funded projects remains more or less stable. The most popular were Academic Networks on entrepreneurship and innovation (four applications received and two funded in 2009). Most of applications (more than 1 per call) came from Greece, Belgium, Italy and UK. The relatively high success rate of the Erasmus networks as compared with other centralised measures of LLP is explained by the limited number of applications. On the one hand, this is due to the specificity and high requirements set for the networking activities, such as the requirement to involve at least one partner from the 31 LLP eligible countries into the network consortium. Most of networks include an average of institutional partners in their consortiums. Therefore, substantial coordination resources as well as previous cooperation experience are required to build up a successful network. Accordingly, the numbers of applications are not so numerous as compared with the other LLP actions. This is a young programme - initially the quality of the proposals was rather low, yet it improved over time. Especially in the beginning the multi-partner requirement (partners from at least 31 countries) was a stumbling block. However, the general idea is to fund less but larger networks. Now the authorities are discussing the option of slightly reducing the multi-partner requirement from In addition, despite the amount of the applications for the Erasmus Networks was increasing by 50% per year from 2007 to 2009 (a slight decrease in 2010), the action is somewhat distant from the final beneficiaries. The promotion of this action and dissemination of its results could be organised in a more effective way. This finding was confirmed by the interviewees at all levels (EU level and national level authorities, institutional beneficiaries). Overall, there is a need to keep a balance between ensuring sustainability of the older academic networks (that are developing new activities) and funding new networks. Some of the networks funded by the EAN have received support from the Socrates on a continuing basis (for example, the project ISEKI FOOD 3 is a food studies academic network that has been funded a third time). Previously funded networks are better acquainted with the administrative procedures, priorities and specific requirements of the action. They are better prepared to build good applications. Therefore, for newly formed and less experienced networks it is much harder to compete. The EACEA is aware of this issue. The renewal of the networks is not a priority anymore in the new call for proposals. One solution to overcome this barrier would be to improve coordination with the national authorities and national agencies. The national authorities interviewed complained that they do not possess any data on the results of centralised actions by country or the results produced by networks. The national authorities would like to possess such data for ensuring the adequate response in fostering the participation or exploitation of the results. In case of the Erasmus Mundus the applicants submit their applications to the national agencies (it is obligatory) simultaneously when submitting it to the EACEA. The same approach should be applied regarding other centralised actions to ensure effective sharing of information. Another solution would be to focus on the preparatory activities systematically provided for applicants in the next programme generation, with the purpose of developing more effective and higher quality networks. For example, the Accompanying Measures could be used as a tool for funding thematic seminars, exploratory clustering meetings of good quality projects and partnerships in the relevant interest area of the proposed network, together with other experts and stakeholders in the field. Thematic seminars could in each case be followed by a longer period of more in-depth preparation, also to be co-funded as part of the Accompanying Measures grant, thereby helping to ensure the solidity and quality of the future network. 94 Systematic data on the satisfaction of the institutional beneficiaries with the results of the projects are not available. However, the interviewed beneficiaries were satisfied with the achievement of objectives of the network. According to them, targets are always achieved even despite the low percentage of European funding because the partners involved in the network are very motivated. 94 Proposals for future programme generation, Working Group B, Partnerships in Education and Training (2010). 129

130 The results of the networks are less tangible than, for example, in multilateral projects that seek to produce a set of well defined outputs. The networks usually seek to exchange information among a large number of partners and best practices in solving complex problems. Hence, the networks aim at developing common strategy, which later on could lead to specific outputs or results. The complexity of the task implies that production of concrete outputs (for e.g. common curricular) may not be feasible. 95 The main results achieved relate to common terminology, new contacts, information and academic knowledge exchange. On the other hand, as previously noted, network projects usually serve as a starting point leading to offspring projects that aim to produce concrete outputs (databases, commercial products, PhD programmes, books or other) What are the main external factors affecting the outputs and results of the sub-programme / action? There are several external factors hampering effective implementation of the EAN action and reducing its popularity. These external factors are however closely linked to the internal (management related) factors of the Programme. Firstly, one pressing issue is related to the selection of good partners. If the institution coordinating a network has no previous cooperation experience, selecting new partners sometimes lead to various problems, such as the sleeping partners problem. The coordinator alone, responsible for the quality of reports and outputs produced, is often left without any tools of controlling the partners: a) the work of people involved in the network is usually an own contribution; b) partners do not bare any financial responsibility. Secondly, language barrier is often a problem. The requirement to involve partners from all the 31 countries was not easy to implement - the experience of all institutions in the network is very different. Sometimes the coordinator needs to produce a terminology book to all the partners, but it still takes time to agree on the common professional terms. Development of common language in certain academic field is, however, one of the main outputs of the network projects as it leads to better understanding throughout Europe and beyond, and thus contributes to the objectives of the EHEA. Thirdly, lack of support from the social-economic partners is one of the external factors hampering implementation of the action. The action sets a requirement to include social partners such as students and industry associations into the project. However, in practice it is quite difficult to motivate these institutions, especially the industry associations, to take part in the project as they do not see a concrete role for themselves in the educational projects. The network projects are perceived as too complex and too distant from the industries that associations represent. It is much easier to reach industry with the FP7 projects that produce concrete research results that can be commercialised immediately. Similar problems relate to motivating student organizations. Fourthly, participation in the networking activities is not sufficiently encouraged by the institutional administrations or the national authorities. While this situation may differ country by country, some of the institutional beneficiaries complained that support from the authorities is more formal than real. The budget only covers part of the expenses, but coordination of networking requires substantial resources. The institutional beneficiaries noted that support provided by the national authorities in encouraging international projects, especially for the new networks, could be more substantial. Without additional support from the national sources institutions without previous administration experience are not motivated to apply to EAN. Economic downturn also had a negative impact due to budget cut-offs, for example in case of national funding for the academic exchanges in some countries (Lithuania) Effectiveness in implementing horizontal policies: How does the EAN contribute to the horizontal priority of multilingualism? 95 Interview with a representative of an EU level institution. 130

131 It may be concluded that the EAN action does contribute to the European horizontal policy of the multilingualism. However, the action does not directly support multilingualism; it is presupposed that the partners in the network should be able to communicate on the academic issues. The networking activities and development of common outputs contribute to the improvement of language skills (usually the most commonly used languages English, German, Spanish, and French) of the participants. However, as noted by the beneficiaries, language barrier often remains a challenge Is the implementation of equal opportunities (EO) satisfactory in the EAN? Does EAN integrate the issues of EO? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? Networks can be evaluated as positively contributing to the horizontal policy of equal opportunities; especially the issues of racial / ethnic origin are relevant in the Erasmus Mundus projects as they involve partners from the third countries. All partners are treated equally despite different religious beliefs, age, gender or racial origin. For example, the main slogan of the interviewed ISEKI FOOD project is join the ISEKI family that proves the idea behind network projects as a family of equal institutions interested in the same outcomes Effectiveness in integrating previous activities into the LLP: Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? The LLP integration had several effects on the contents, management of the Erasmus academic networks and satisfaction of the beneficiaries: - The priorities and scope of the centralised Erasmus actions increased: a larger number of different types of activities were added after the launch of the LLP. After the launch of LLP, Erasmus funds a smaller number of networks projects, but they are substantially larger. The idea was to increase the quality of the management of the centralised measures. - Integration created preconditions for monitoring and avoiding duplications: before the integration it was virtually impossible to check whether different programmes fund the same organizations, which implement similar types of actions. After the launch of the LLP it theoretically became possible track such potential duplications. On the other hand, the Executive Agency noticed that an effective system is still needed that would allow tracking the potential duplications of EU funding for example when the same results are being funded by the Erasmus networks or the multilateral projects. - One of the main effects on the management of LLP was the standardisation of administrative forms that should have lead to smoother administration of the programme. Still, according to some interviewees the Executive Agency has taken a rather narrowly administrative point of view in standardising the forms. In case of Erasmus networks it is already clear that standardisation has lead to negative effects due to the incompatibility of the required format and the specificity of the projects. Now the discussion has been started on how to develop a special simplified version for networks projects, and, according to the EACEA, first steps have been taken to improve the administrative documents especially for networks projects. Overall, as previously noted (section 2.3), the idea of having separate management forms for networks has lead to the discussion on further reducing the barriers between the sub-programmes in the period. After the integration of the LLP and the establishment of the EACEA the institutional beneficiaries also noticed that the management process of the programme is organised in a more effective way as compared with the TAO s How successful is the EAN in reaching the target groups and institutional beneficiaries of subprogrammes? According to the data collected for the case study analysis, the success of Erasmus Academic Networks action in reaching the target groups could be improved. On the one hand, more applications are submitted than funded and given the specificity of the action this is a good result. However, the interview results confirm the fear that the information on the Erasmus 131

132 centralised actions does not effectively reach the potential target groups at the national level. According to the institutional beneficiaries, the information on the centralised actions is usually disseminated from mouth to mouth. Apparently, those potential beneficiaries that are not part of older networks might not be reached via the information dissemination campaign at the European level. The visibility and promotion of networks, especially at the national level, should be improved. The EACEA considers cooperation with the National Agencies in the dissemination and promotion of the centralised LLP actions of crucial importance. Additional money has been granted for the participation of the National Agencies in the meetings of coordinators. The National Agencies have developed efficient information dissemination platforms that could be used for sharing information on the centralised measures as well. More active cooperation between the EACEA and National Agencies has to be ensured in the future periods of LLP implementation. This specific issue is already under discussion with the DG EAC. As also noted by the EU level interviewees, cooperation with National Agencies is more necessary for the general promotion of the Lifelong Learning Programme than for reaching project target groups (this depends greatly on the type of project/activity). Overall, the beneficiaries felt that Erasmus academic networks are poorly recognised and much less visible than, for example, the Erasmus Mundus networks. It seems to them that EAN is not a priority action as compared for example with Erasmus Mundus To what extent are the EAN s results adequately disseminated and exploited (at both horizontal and vertical levels 96 )? Overall, despite substantial efforts are devoted for the dissemination of the results, better follow-up and long-term monitoring of results is necessary to be established in the future periods of the LLP management. Dissemination and exploitation (valorisation) of the results of Erasmus networks is an issue that receives a lot of attention from both the EACEA and the DG EAC. The EACEA monitors the dissemination and valorisation of network s results during the lifespan of the project, while the Commission is responsible for further analysis and mainstreaming of results into the policy making. The project coordinators (institutional beneficiaries) commit to ensure the sound dissemination of results by providing a results valorisation plan in the project application. Beneficiaries have to report all dissemination activities in the final report of the project, which are assessed by experts. The results of the evaluation have a direct impact on the final payment of the project. For example, a very low score means a substantial reduction of the reimbursement of staff costs. Dissemination and valorisation of results are part of the award criteria of the selection as well as criteria for the assessment of the reports (final and interim). In the project application and project reports an assessment of the sustainability of the project further the end of the contractual relation with the EACEA is performed. As a minimum, each network is expected to carry out the following operational activities: a) Establish a website and other appropriate tools to support information exchange and dissemination; b) Provide the players in Erasmus with full information about the network s events and activities; c) Organise an annual meeting in the thematic area of the network. The meeting may also bring together representatives of other Erasmus-supported activities in the field concerned, including notably multilateral projects and intensive programmes. It can take the form of an open seminar or conference, thereby encouraging collaboration between Erasmus-supported projects and other relevant initiatives; d) Take appropriate measures regarding the evaluation of the network's performance. The EACEA implements numerous activities to ensure wide dissemination of the results and best practices of the funded projects. Project coordinators participate in 2 coordinators meetings during the life time of the project (a kick-off meeting during the first year and a follow-up meeting during the second year). Every year, thematic cluster meetings are also organised. The EACEA receives the final results of the projects through the reporting provided by the beneficiaries and provide the Commission with feedback on programme results to integrate these into the work of Commission. Experts assessing the implementation of the project carefully evaluate the level to which the valorisation of results was implemented. The experts hired by the EACEA at 96 Horizontal mainstreaming refers to the project results and good practices adopted by other stakeholders. Vertical mainstreaming refers to the integration of the good practices and project results into general policy on national and/or European level. 132

133 different stages of the project (interim and final reports) also assess the contents of the concrete outputs and results produced by the projects (such as books, new curricula/teaching modules, introduction of TUNING methodology, quality assurance methodology, conference proceedings, website / e-learning modules, etc.). The EACEA does not monitor exploitation of project results after the end of contracts, but assists the DG EAC in selecting best practice outputs and results. The Commission selects best practice outputs and results, establishes expert groups to discuss results of the actions. The forum for Enterprise University, which was initiated by the DG EAC, facilitates mainstreaming of the main results of this action. The Additional measures should also contribute to ensuring the follow-up. Nonetheless, the exploitation of the networks results is rather difficult: a) Networks work on implementation of Bologna goals in one sector and it is rather problematic to transfer the results to another sector. b) Networks are set up for EU policy implementation (clusters on higher education are funded), but also they are expected to influence policy making. Exploitation of the results at the national level is hampered by the lack of support from national policy makers and inability to involve them into the projects. Involvement of the policy makers into network s activity is crucial for achieving this objective. c) The sustainability of the networks without European funding is problematic. Without funding for meetings of partners virtual network would not last - this was confirmed at the interviews with the beneficiaries. Interviews also indicated positive examples of the exploitation of networks results. One of main effects of networks is that they increase participation in other international programmes and actions, such as the Framework programme, Erasmus student mobility etc. Some interviews with the beneficiaries also confirmed the positive effect of networks results on policy at the national level. For example, the knowledge of Portuguese experts involved in the network was utilised by the national authorities in the process of establishing the national system of credits. The institutional beneficiaries perceived the dissemination of project results as successful. The main target groups (teachers and academics) are easily reached because the outputs are disseminated through the global platforms such as the academic databases (e.g. Springer), discussion platforms, and conferences. The project interviewed has served as basis for starting the Erasmus Mundus project that now involves institutions from all over the world (Asia, Australia) that allows disseminating the project outcomes on global scale. The projects use all types of instruments for dissemination, such as: Internet websites; manuals, readers, other material; handbooks, books; studies, reports, other publications; seminars, conferences, workshops, other events; video/internet conferencing; newsletter; TV; radio and newspaper. One of main tools are databases where the network partners share information on the research infrastructure (laboratories, tools, equipment) available in their institutions, on the study and research programmes, pictures and videos of new products, etc. The databases are highly efficient in searching for relevant information, starting new academic or research projects, study programmes. However with the end of the project the interviewed beneficiaries are planning to make the database commercial, i.e. the usage will be paid to ensure the resources required for updating the data. 4. EFFICIENCY 4.4. Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at all levels: the Commission s, the Executive Agency and the beneficiaries level)? Overall, data collected for this case study indicates relative efficiency of the EAN action. However, there are still some gaps in the current management and control system that are described in a section below. Gaps identified relate to: a) The harmonisation of the application forms and other management documents; b) Insufficient funding of the EAN that leads to difficult management of the projects at the beneficiaries level. 133

134 In the period from 2007 to 2010 the management and control system of the LLP was heavily reformed. Firstly, during 2007 the Executive Agency focussed on harmonising the application forms and procedures. This resulted in a single application form for all multilateral projects, networks, accompanying measures and studies and comparative research. The forms used for other centralised selections used the harmonised model as a base to develop the specific forms required for the remaining actions. Secondly, the database developed to manage the Socrates Programme (SAYKISS) was enlarged to house the new actions within the LLP. Thirdly, the Agency undertook a thorough revision of the different approaches to selection and developed a new system and scoring mechanism that could be used across the sub-programmes and Transversal Programme. The aim of harmonising the system was to simplify the management of sub-programmes for both the managing institutions and the beneficiaries, and thus to reduce the red tape. However, it seems this approach is not efficient for the Erasmus networks projects. According to the interviewees, the harmonised administrative forms were less suitable for networks because these projects differ greatly from other groups of projects (they encompass many more partners). The institutional beneficiaries interviewed confirmed that the application forms have become more complex with the introduction of the LLP. It was also hard for the experts to evaluate contents of the applications (questions were not suitable for this purpose). In 2009 a group of experts reflected on the simplification and harmonisation of the administrative forms. Reasons described above indicated the need to develop a separate application form for networks. According to the Executive Agency, steps are taken to develop a new form separately for networks from the new call for proposals, containing two parts: the administrative data (directly transferred to the EACEA database) and the project content-based. There is a need for a tool that would lead to more flexibility. Overall, interviewed beneficiaries are mostly satisfied with their cooperation with the Executive Agency. The administration of the programme has much improved as compared with the TAOs. The institutional beneficiaries and the Executive Agency usually communicate by . The support and responses by the Agency is considered professional and effective. Sometimes there are small mistakes in the applications or reporting forms, or misunderstandings in filling the databases of monitoring data, but these are minor problems that do not create much administrative load to fix them. The complaints expressed by the beneficiaries as well as the authorities at the national level relate to the fact that the Executive Agency has taken an administrative point of view, a bit distant from the action. According to the interviewees, the cooperation should be more dialogue-based (not control-based). Currently the focus is on small details (the application and reporting forms changed often, but changes are insignificant) instead of focusing on the contents and results of sub-programmes. By devoting attention on the administrative details, important resources are wasted. The national authorities also called for dialoguebased programme management. The cooperation between the EACEA and the Commission is perceived as good and productive. The roles are rather well defined; the EC provides timely and adequate guidance to the Agency. However, feedback from the Agency on issues related to policy implementation should be envisaged in near future. Difficulties to manage projects at the beneficiaries level also relate to the insufficient budget. As described above, total European funding is sometimes only up to 35-40% of the total budget. This funding covers the costs of travelling, publishing, building databases, but the beneficiaries mostly cover the man-days of persons involved from their own sources. This leads to lack of tools for controlling the work of partners, and to losses in programme results quality. As noted by the beneficiaries, current financial management system does not encourage large projects (contrariwise to the political priorities expressed less but larger projects ) because the upper limits of funding (not more than 60% of the total budget and not more than 200,000 per year) do not take into account the numbers of partners. Hence, the more partners involved in a project, the more difficult the coordination, but the less budget per partner. According to the beneficiaries, with current financial provisions it is not efficient to have more than 40 partners. This should be taken into account in the future periods of the action s implementation. 134

135 4.5. Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? Overall, it may be concluded that available management resources are not sufficient for improving the quality of the programme. The Executive Agency has undergone staff fluctuations after its establishment. Currently the number of staff at the Agency is stable, and the available management resources are considered appropriate and sufficient in view of the objectives and tasks allocated to the Agency. External expertise is used to assist in the process that requires content-related knowledge, e.g. when evaluating project results, in the framework of in situ visits, cluster meetings. Pool of experts is still evolving while it takes time to find and train competent experts in specific areas. A 30% renewal of experts is required every year in order to bring new ideas and expertise on board. Periodical training and ongoing supervision of external experts is ensured for building the required capacity and quality of the evaluation. However, the available management resources might not be sufficient for implementing the dialogue-based administration of the programme as requested by the beneficiaries, the national agencies and the national authorities. The EACEA mostly deals with administrative matters; while the DG EAC has the structure internally that allows evaluation of programme s effects. The problems are related to the resources available for performing qualitative analysis of the available data at the DG EAC. The feedback from the Executive Agency usually relates to the quantitative results of the actions. Previously there were separate units for programme and policy making. The new Director General has merged the units to allow better feedback to the policy making. However, the policy units (in the higher education sector) are understaffed and do not have sufficient capacity for evaluation and monitoring of long-term effects. The institutional beneficiaries (the finding mostly relates to the coordinators of networks projects) thought that resources for the management of the networks are not sufficient, because of the workload related to coordination of the pool of partners (often more than 100 partners in the network). They noted that administrative work of the coordinators should be reduced. First of all, the work when dealing with large numbers of partners could be better balanced (as described above, the financial provisions are too modest). Also, administrative workload could be reduced by providing some management tool/-s for monitoring work done by partners online. The interviewees thought that the EACEA could develop some management tools available for all networks to use. Other issues relate to the financial administration of the programme: a) introduction of the lump sum as used to be in the previous programme; b) introducing some type of agreement between the coordinator and the partners could improve communication, and partners would feel the ownership of the project Is the monitoring information sufficient, is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? Overall, the monitoring system is considered as sufficient, but the level of qualitative analysis of data as well as extent of its use for decision making could be improved. One of the most addressed criticisms towards the monitoring system was that it focused too much on the administrative and financial information. At the same time the administrative resources for collecting and systematising information on outputs and results was missing. According to the interviewees 97, currently the approach has changed. The Agency monitors achievement of outputs and results via three instruments: - The experts check the contents of the applications and (interim and final) reports. As a result, attainment of planned outputs and results is monitored throughout the lifecycle of the projects. - The Agency organises visits to assess the progress of each project. - The Agency organises thematic meetings of organisations, which implement projects in similar areas. These meetings are used for presentation of key outputs and results of the relevant projects. 97 These data was collected during the EU level interviews visit in April

136 The monitoring mechanisms are considered effective and roles are clear in regard to transmission of information, communication and dissemination of project results to the Commission. There are official procedures for feedback and exploitation of results between the EACEA and DG EAC. The EACEA aims to supply the Commission with tailor-made data in the form of good practice, success stories and key figures for visibility and valorisation purposes (also on certain thematic issues). For example, the EACEA has created country fiches by country and by selection year that contain various monitoring data. This information is highly useful for examining country participation results and is used in the decision making process, e.g. when making decisions on the priorities of next calls for proposals, for internal reflection, communication, briefings etc. On the other hand, the Commission admits that resources available for the qualitative analysis of data are not sufficient and more attention could be devoted for the monitoring and evaluation of the long-term results. The national authorities on their side also complained about the lack of data coming from the EU level authorities on the results of their country s participation in the centralised LLP actions. Hence, the analysis, usage and sharing of the monitoring results could be improved. 136

137 National case studies 7. Erasmus Student Mobility for Studies (Outgoing Students): Lithuania 1. INTRODUCTION An overall purpose of this case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme s action concerned. The case study elaborates only on the outgoing students mobility; therefore data was not collected on the incoming mobility or the students placements in private companies. The complementary case study on the incoming students mobility in Germany will contribute to the exploring of the full picture of the Erasmus students mobility for studies cycle. The information presented in this case study is based on the following sources: - In-depth interviews with the national agency, the national authorities and the institutional beneficiaries of the Erasmus Student Mobility for Studies action. In total nine persons have been interviewed at eight interviews (the full list of interviews is presented in the Annexes); - Desk research, which sought to analyse national legal acts, action s implementation statistics and other documentation, which was provided by the national agency, the national authorities and the institutional beneficiaries. The interviews and desk research were carried out during the period of 20 March 20 April RELEVANCE The main objective of this section is to describe the European Added Value of the selected action To what extent the intermediate and specific objectives of the action remain pertinent to the national policy priorities? The primary objective of the Erasmus Students Mobility for Studies (SMS) action is to foster mobility of students in order to: 1) enable students to benefit educationally, linguistically and culturally from the experience of learning in other European countries; 2) promote cooperation between institutions and to enrich the educational environment of host institutions; 3) contribute to the development of a pool of wellqualified, open-minded and internationally experienced young people as future professionals in the national economy. Overall, these objectives remain highly pertinent to the national policy priorities. There is a high level of coherency between the objectives of Erasmus SMS and the long-term objectives of the national higher education policy 98 in Lithuania. The strategic and operational policy documents highlight the need to support academic exchange as a driver for competitiveness of the national higher education system. It is widely acknowledged that the Erasmus sub-programme has served as an external stimulus for opening and constantly upgrading the study programmes in order to achieve higher European standards. Increasingly the internationalisation of higher education is supported by the national policy: - The Programme on Fostering Internationalisation of Higher Education approved in 2008 outlines the national measures foreseen for fostering students mobility. The overall target is that each student in Lithuania has at least some experience of international mobility in his studies cycle. - National budget s co-financing for outgoing students mobility is provided each year up to 100% of the Erasmus SMS support. From 2010 the co-financing is also foreseen from the funds of EU structural assistance for The percentage of the co-financing provided from the national budget for 98 Outlined in the National Long-term Development Strategy (2009), the Law on Education and Science (2009), the Government s Programme (2008) and the Programme on Fostering Internationalisation of Higher Education (2008). 137

138 the Erasmus students mobility in Lithuania is highest among all 31 participant countries (it constituted up to two thirds of the student mobility grant before the accession to the EU). - Up to now the number of outgoing students was one of the indicators in the formula for calculating the State s funding for the national higher education institutions. Recently the attention of the policymakers has shifted towards achieving the balance of incoming and outgoing academic mobility. The State applies the indicators measuring numbers of the incoming students and balance of incoming/outgoing mobility both when calculating block grants to higher education institutions and the co-financing for Erasmus SMS. The fact that the national stakeholders are relatively influential in setting the Erasmus grant award criteria contributes to strengthening of the bottom-up links and synergies between national and EU level priorities. The criteria set at the national level allow: a) reflecting directly the national priorities in the field of students mobility 99 ; b) putting more emphasis on qualitative indicators while the EU is mainly interested in creating critical mass. The grant award criteria are set by the Commission of experts responsible for coordination of international EU programmes To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? Two main target groups benefit from the support of the Erasmus SMS action: students (as direct individual beneficiaries), and the higher education institutions awarded the European University Charter (as indirect institutional beneficiaries). According to the data collected for the purposes of this case study, the Erasmus Students Mobility for Studies action remains highly pertinent to the needs of both target groups in Lithuania. The institutional beneficiaries indicated both positive and negative effects of the Erasmus SMS. From the one hand, the Erasmus support mobility for students contributes to the brain drain from countries with less competitive higher education system. The most promising students often choose Master studies abroad after their Erasmus experience, hence the institutions lose potential prospective students. On the other hand, the effect that the Erasmus incoming and outgoing students mobility has on the upgrading of the study programmes and overall study process in Lithuanian higher education institutions has not reached the saturation point yet. The main hypothesis behind this statement is that the more advanced and international the study programme, the less benefit it gets from the outgoing students mobility. 100 Though the effect of the Erasmus on the development of the internationalisation strategies at the higher education institutions has been substantial, there is still a huge gap to catch in terms of the quality of study programmes. The interviewed institutions admit that participation in the Erasmus programme serves as a perfect criterion for measuring the international competitiveness of the institution and its faculties or separate study programmes. The interviews with the individual beneficiaries were not carried out for the purposes of this case study. However, several indicators allow concluding that the programme remains pertinent to the needs of the individual beneficiaries. Firstly, the numbers of student applications for the mobility grant had been increasing each year. Due to the lack of funding, some of the institutions only fund from 30 to 50% of applications. Secondly, several proxies allow concluding that the programme is having a positive effect on the students skills and career prospects: a) though systematic data are not available, the higher education institutions notice improvement in the social skills of the Erasmus participants; b) in the discussions with higher education institutions, employers prefer graduates with the international experience as potential employees. Thirdly, vast majority (about 90%) of outgoing Erasmus students are highly satisfied with their experience and are willing to repeat it. 99 The calculation of the grants for participating institutions is based on the following criteria: 1) the total number of students (excluding the 1st year students involved in undergraduate studies) at the higher education institutionhigher education institution or the total number of teachers /staff; 2) the rate of higher education institutionhigher education institution s outgoing mobility past performance of last two years; 3) the rate of number of outgoing and incoming students at the higher education institution higher education institution(bidirectional mobility); 4) the rate of the geographical distribution of outgoing students or outgoing teachers; 5) the rate of number of outgoing students by the subject area; 6) planned mobility numbers (demand) for the new actions (placements and staff training) were taken into account as well. Source: the National Report by the NA, Based on interviews with the National Authorities. 138

139 2.3. The European added value. Would other national schemes / instruments provide enough support to student mobility related activities, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? Information collected during this case study indicates that the European added value is substantial. The added value is created in several interrelated areas. Firstly, in the absence of the LLP the likely level of funding provided by the national authorities to student mobility related activities in Lithuania would amount to about half of the current funding. Accordingly, only half of the current outputs would be reached. The main reasons behind the inability to reach the same students mobility results with other sources relate to a) the lack of alternative mobility funding sources and b) low capacity of the student families to co-finance studying abroad. Erasmus is the main source of support for students mobility in Lithuania. Other smaller sources exist (such as the Nordplus programme by the Northern Ministers Council, grants offered under the bilateral country agreements or the loans for studies offered by the National Studies Foundation), but the capacity of these sources altogether is very low. Secondly, the Erasmus programme has served as one of the main drivers for the internationalisation of the Lithuanian higher education and thus contributed to the international visibility and competitiveness of the Lithuanian higher education. This would not be achieved without the support by the Erasmus as internationalisation of higher education for a long time was not among the national policy priorities. The added value is relatively high in Lithuania compared with the Western European countries having several funding sources for students mobility. The popularity of the Erasmus in this country can be also explained by the relatively large quality-related differences in the higher education systems in different European countries. While the students from Great Britain or France might be more interested in studying in Asia, Australia or the USA, the students from Central and Eastern European countries do value and highly benefit from the international experience in the European Union What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? No obvious cases of duplication of efforts were identified during this case study analysis. In case of the Erasmus Students Mobility for Studies action, the national stakeholders only identified these synergies created by the integration of LLP: - Firstly, the transfer of student placements from the Leonardo sub-programme to the Erasmus subprogramme is evaluated as a very effective solution that has allowed integrating the study process (study and placements) into one cycle. - Secondly, there are synergies related to the more systematic approach towards the administration of the LLP, mostly related to the information provision and reporting. Integrating the previous programmes into one Lifelong Learning Programme has allowed better reach of the target groups and more efficient use of the administration resources at the NA. The preconditions for the coherency of efforts at the national level were created, firstly, by establishing a separate agency for coordination of the LLP the Education Exchanges Support Foundation. Secondly, the national Commission of Experts responsible for coordination of the international EU programme serves as a national-level coordination instrument that aims to assure that the implementation of sub-programmes is coherent. On the other hand, there is no evidence that would point to the effectiveness of these coordination instruments, except for the obvious cost-efficiency in using the resources for the administration of the programme. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? 139

140 Overall, the progress towards achieving the national student mobility targets in Lithuania over the period allows indicating high effectiveness of the action s implementation: - Firstly, the funds of the action are used up to 100% each year. The number of outgoing students had been increasing approximately 10-15% per year until the 2008/2009 academic year results when outgoing students mobility faced only ~1,4% increase. - Secondly, according to the institutions' and students' reports vast majority of the outgoing Erasmus students (more than 90%) were highly satisfied with the exchange experience and were willing to repeat it. 101 Based on the information collected during the case study, we make a conclusion that the main factor that directly influences the outputs achieved by of the Erasmus SMS is the level of funding. Moreover, the current level of outgoing student mobility in Lithuania is achieved only due to the hard budget-saving strategy at the institutional beneficiaries. The interview programme indicated that both EU and national funds only satisfy from 30 to 50% of the mobility demand. During the 2008/2009 academic year only about 1,14% of all students enjoyed the Erasmus mobility experience, which is a good result compared with other countries, but a bad result given the institutional targets. Most of annual targets for outgoing students mobility at the interviewed higher education institutions were not achieved due to the lack of funding. For example, one of the universities has an annual target to send abroad 5% of all students. None of the faculties had achieved this target so far. During the last two years the national co-funding of the Erasmus programme was cut down due to the national budget crisis. Responding to the shortage of funds for meeting the annual demand, a typical solution in all the interviewed institutions was to boost the outputs by cutting the scholarships and the duration of studies abroad to the minimum. 102 While this helps keeping the level of outputs from dropping down, the situation is unfavourable for the individual beneficiaries, especially the ones from socially disadvantaged families. While the quantitative outputs of the Erasmus SMS are pretty straightforward (i.e. numbers of mobility), it is immensely difficult to provide estimates of the achieved results of the outgoing Erasmus students mobility for studies, partly because the results of the Erasmus SMS are not clearly defined and not evaluated at the national level. These results can be grouped according to the institutional and individual levels of the LLP programme. The results of the programme at the institutional level can be grouped under two broad categories: a) the internationalisation (direct) effect; b) the quality of studies (indirect effect). Firstly, the institutions interviewed claimed that participation in this action mostly had a positive effect on the internationalisation and international visibility. For example, positive effect was achieved on establishing or enlarging the network of partners or contacts, improvement of the international profile, greater awareness of European dimension of higher education, and changes in the administration and management practices within the institution (related to internationalisation). Especially after the national measures 103 on strengthening the balance between incoming and outgoing students were introduced, the situation has improved dramatically. Over several last years the ratio of incoming/outgoing students has changed from 1/10 to 1/2. Accordingly, this had a huge impact on the internationalisation strategies of the higher education institutions pressure was put on developing international study programmes, assistance services for incoming foreign students, the mentors services, and the recognition of study periods. This has an extensive contribution on the modernisation of education policy and management strategy within the institutions. Secondly, there were positive indirect effects received through the feedback of students with study abroad experience. Such effects are noticed on the broadening and deepening the range and content of study programmes, courses or teaching modules, and better understanding of the higher education systems in other countries. The results of the Erasmus students mobility at the individual level relate to the improved social skills, language skills and overall entrepreneurship of the individual. Such individuals are more active and better 101 Sources: student mobility statistics provided by the National Agency at < >; the National Reports provided by the NA to the European Commission. 102 The minimal requirement of studies abroad set by the European Commission is 3 months. Minimal requirements for the monthly study scholarships according to different groups of countries approved by the Minister of Education of LR (minimal scholarship is 350 Euros/month). 103 Such as making the balance of incoming/outgoing students an indicator in allocating the Erasmus funds for students mobility for studies as well as the national institutional funding (block grant). 140

141 evaluated at the labour market (this judgment is based on the discussions of the institutions administration with the employers during career days and similar events). However, no evidence within this case study was found confirming that the study abroad improves the academic performance of the students. Overall finding is that the LLP Erasmus Student Mobility for Studies action has contributed both to the accessibility of higher education (i.e. benefits were created to the individuals in terms of academic knowledge and social skills acquired) and the quality of higher education (this effect is stronger from the incoming student mobility and student placements) What are the main external factors affecting the outputs and results of the action? Desk research and interviews indicate that main factors that positively affect the outputs and results of the Students Mobility for Studies action relate to a) the level of development of both national economy (the overall social and economic situation and the standards of living), and b) level of development and competitiveness of the national education and training system. While the standards of living and the economic and social development level in the country has not reached the European average, many students see the international study experience as their chance for better future professional or academic prospects (possibly in another country). Therefore, numbers of students willing to study abroad have increased with the economic downturn in Lithuania. A significant part of the prospective students later on choose their Master or PhD level studies abroad in the more prestigious universities (as compared with the national ones). Hence, the increase in the outgoing student mobility strongly correlates to the economic migration processes. The outputs of the Erasmus SMS directly mostly depend on the ability to co-finance the Erasmus SMS grant (by national funds, by means of students and their families). Hence, the external factors negatively affecting the outputs of the action during the 2007/2009 period can be evaluated according to the effect they had on the funding of the programme. These factors can be grouped under such headings: - Firstly, economic factors, such as the economic downturn that has sharpened the problems related to the economic status of the country (living standards in Lithuania as compared with wealthier European countries) and the related socio-economic situation of student families; - Secondly, particularities of the national education policy, namely the higher education reform implemented in The higher education reform has introduced the student voucher system from the academic year of 2009/2010 and shifted the attention of political leaders from all matters including the international EU programmes towards the challenges raised by reform. - Finally, there are also some small mobility barriers related to the lack of facilitation mechanisms in the hosting countries / institutions, but these obstacles are less significant and easily solved. The positive or negative effects of the higher education reform had not yet fully revealed themselves. The student voucher system was introduced in 2009, so today s 1 st year students will only be eligible for the Erasmus mobility support in the academic year 2010/2011. However, the interview respondents already forecast the decrease of attractiveness of the Erasmus mobility grants. After the introduction of vouchers more students have to pay for studies, and together with the effects of the economic and financial crisis this makes the co-funding of the Erasmus grant by students and their families less probable. Nonetheless, there is a possibility that those higher education institutions that were more competitive and attracted more vouchers would be able to free some resources for complementary co-financing of the Erasmus grants awarded to prospective students from socially disadvantaged families. The economic downturn that manifested itself in Lithuania over the period has resulted in the national budget crisis. This has lead to the reduction of the national co-funding for the Erasmus mobility grants, and accordingly to minimising the awarded scholarships as described above. Nevertheless, these factors in their turn had a direct negative effect on the numbers of outgoing students. The economic downturn worsened the financial situation of students families and their ability to co-finance the mobility grant. Also, fewer students are willing to take State-granted loans to fund study abroad periods. Hence, lack of funds remains the largest obstacle to mobility in Lithuania. The scholarships provided by Erasmus only cover part of the study expenses; hence some of the bright students abort their intentions for study abroad for the economic reasons. Therefore, despite the fact that more students are willing to study abroad and see it as an opportunity for better academic or professional career prospects, fewer students can actually afford international mobility in the period of economic downturn. The numbers of students willing to 141

142 study abroad with limited or null scholarship are minor and decreasing. Some of the institutions have small funds for supporting students from socially disadvantaged families, but the amount of funds depends on the overall financial situation of the institution (i.e. is very limited). Lack of funds also determines the duration of studies (short studies are preferred) as well as the countries of host institutions (countries with relatively lower standards of livings are more popular). The forecast for the 2009/2010 is that student mobility numbers will be falling due to the decrease in the national funding and the overall economic situation. And vice versa, higher outputs of students mobility could be easily reached with injections of additional funds. The reserve for achieving similar quantitative mobility outputs (number of outgoing students) with existing funds lies in sacrificing the quality of mobility periods (i.e. eliminating the countries with higher living standard, longer than minimal study periods, more expensive study programmes, etc.). Other smaller mobility barriers indicated refer to: a) difficulties caused by unexpected changes in the agreed study programmes or other study conditions, which led to problems of study period recognition by the home institutions; b) difficulties in receiving transcripts of records, documents proving the mobility period from host institutions on time. The latter problem, as mentioned by the interviewees, most often relates to the host institutions from Switzerland (the silent partner of Erasmus). In general, these barriers are not recognised as major obstacles as all the problems are relatively easily solved. Some of the external factors relate to the values and attitudes of students and lecturers on the one hand, and to the differences of the quality of education systems in the home and host institutions on the other. For example, students with good academic results sometimes reject the Erasmus mobility option because of the fear to spoil their academic grades and to lose the State-funded scholarship. Mobility results at the level of faculties also depend on the position taken by the professors (if they encourage academic exchanges). While some faculties are very positive towards students mobility, other still react negatively to the higher requirements raised by students with experience of studying abroad, and it can be hard for such students to integrate back into the national higher education system Effectiveness in implementing horizontal policies How does the action contribute to the horizontal priority of multilingualism? Overall it may be concluded that the Erasmus students Mobility for Studies action contributes directly to the multilingualism objective by improving the language skills of its participants. The horizontal priority of multilingualism is integrated into the calls for proposals: - On the one hand, the priority is given to students that have good knowledge of the national language (hence, not only the English language skills are prioritised); - On the other, additional one-month Erasmus scholarship is provided to those students that get support for the intensive language courses as an additional incentive for improving the language skills. Majority of the individual beneficiaries report that their foreign languages skills have improved after the study period abroad. According to the monitoring information stored by the National Agency, English is not the only study language in the Erasmus programme other dominant languages of studies abroad by the Lithuanian students are Spanish, French and German. The National Agency keeps a database with the information on each student including their language proficiency and the language of studies during the mobility visit. The information collected during the case study analysis suggests that both integrated and differentiated approaches to encouraging multilingualism appear to be effective at the national level, whilst the respondents were not aware of any projects supported by the Transversal centralised action of Languages. While the Lithuanian higher education students have relatively good language skills, the language skills of the beneficiaries of the other programmes remain a big problem. For example, very is a huge need for the decentralised measures for encouraging multilingualism in the sub-programmes of Comenius and Leonardo. The National Agency indicated many cases when the beneficiaries took interpreters to the study visits (despite these type of expenses being ineligible). More effect would be achieved if there were decentralised measures for developing languages skills in the Comenius and Grundtvig, Leonardo sub-programmes at the national level How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? 142

143 Overall, the issues of equal opportunities are implemented on average. More systematic approach should be taken towards implementing the equal rights of students from the disadvantaged families to take part in the Erasmus Students Mobility for Studies. Integrated approach 104 is implemented in the case of ensuring gender equality by delegating the responsibility to ensure gender equality to the institutional beneficiaries. On the one hand, equal rights of men and women are not prioritised in the Erasmus Students Mobility for Studies in Lithuania because of the common practice to treat women and men equally the grant is awarded only according to the academic results of the student and other criteria. On the other hand, higher education institutions are obliged to report to the National Authorities on how the equal opportunities (respecting equal involvement of men and women, disabled students, as well as students belonging to national minorities) were ensured during their accountability process. In case of disability, on the contrary, the differentiated approach is applied. From 2008, the Erasmus Students Mobility for Studies action provides additional funding for disabled students. Disabled students can apply for an additional scholarship covering the necessary medical expenses or the costs of the attendant person. All students apply directly to the NA; trilateral grant agreements between the NA, sending institution and individual participant are signed and the grants are paid directly to beneficiaries. According to the data provided, the popularity of this complementary measure is low (only seven students were awarded additional grants over the periods of 2008/2009 and 2009/2010). The unattractiveness of this measure might be either related to the low demand of the measure in general, or to the specific attitudes of the beneficiaries. According to an Erasmus coordinator at a higher education institution, disabled students feel inconvenient if they have to highlight their disability while applying for the scholarships. No suggestions on how this issue could be treated were provided during the interviews. An insufficiently addressed problem relates to the mobility of the students from socially disadvantaged families. As it was noted earlier, these students have fewer possibilities to take advantage of the Erasmus mobility grants. The grants do not cover full expenses of the studies and in Lithuania, where living standards are still relatively lower than in many older EU Member States, so this is a significant problem. No priority is applied at the national level that would provide support for the students from socially disadvantaged families, except for the possibilities to get a privileged loan through a National Studies Foundation. Separate higher education institutions have their own small funds for supporting the socially disadvantaged students that demonstrate best academic results. The National Agency mostly addresses this issue by organising various thematic events trying to attract the attention of society and the policymakers. For example, this year is the Social Integration Year; accordingly the thematic event will be organised on the issues of social integration (related to the implementation of various LLP sub-programmes and actions) Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? Overall, judging from the data collected under this case study analysis, the advantages of the integration of the previous activities into the Lifelong Learning Programme outweigh the disadvantages created. There is sufficient evidence that the integration of the LLP was successfully implemented at the national level. However, integration of the LLP at the administrative level still remains a challenge. The effectiveness of the integration of the LLP in this case study can be evaluated in terms of: - The benefits created for the beneficiaries that otherwise could not be achieved (content-related benefit); - The administrative simplification or any other management related added value that had effect on the efficiency of the LLP, for example has loosened the resources at the national agency or the institutional beneficiaries. 104 Under the differentiated approach special actions and projects of the LLP could be designed and implemented to deal with specific horizontal policies. Under the integrated approach the Community horizontal policies could be integrated in the different sub-programmes and actions, depending on their priorities. 143

144 The major content-related benefit (related to Erasmus SMS) of the integration of the previous activities into the LLP is undoubtedly the transfer of the students placements from the Leonardo sub-programme to Erasmus. This transfer has allowed creating the continuous study cycle for the studies and created synergies between the higher education studies and first professional experience of students. Integration of this activity into the Erasmus sub-programme had other positive effects: a) made the administration of both student mobility for studies and student placements easier under one roof of Erasmus; b) had an impact on the overall study process by increasing the duration and importance of student traineeships in companies (previously those placements used to be much shorter); hence, the whole study cycle was adjusted to the Erasmus requirements; c) has fostered the university-business cooperation processes. The other advantages of the LLP integration relate to the management-related benefits for the decentralised actions on both national and institutional levels: - Firstly, the integration has allowed reaching wider target groups with lower cost and has reduced the duplication of efforts in disseminating the information about the separate sub-programmes of the LLP. The integration of the LLP at the national level has been fully implemented: the National Agency operates as a single one stop shop for all sub-programmes, and this has allowed for better coordination, dissemination of results and other information and created cost-efficiency in the administration of the sub-programmes. - Secondly, the integration of the LLP, according to the interviewees, has encouraged closer and more direct involvement of the national authorities into the programming and implementation of the programme. Before the establishment of the LLP, the administration of the international EU programmes was strictly delegated to the national implementation agencies and the involvement of the policy-making institutions was minor. The current structure encourages closer involvement of the Ministry of Education of Science, which is potentially more favourable for the coordination of the EU s and national priorities and the exploitation of the LLP results. However, involvement of the policymakers is not yet effective for achieving sufficient mainstreaming of the programme s results vertically into the policymaking. However, the harmonisation in the procedures and the administrative simplification of the LLP at the supranational level is still an ongoing process. On the one hand, the institutional beneficiaries agree that the administrative process has become easier and many of the difficulties had been solved over the period. On the other hand, they call for greater harmonisation of the application and reporting procedures. The remaining problems relate to the administrative documents such as the application forms and final reports being systematically late and to the overdue payments (see the detailed comments in the fourth chapter of the case study). According to the National Agency, the remaining procedural and administrative integration difficulties are related to the lack of integration at the European Commission s level. There are cases when the EC units provide contradicting answers; due to the lack of cooperation between the units the explanations of the problematic issues take too long. Some of the issues related to the application and reporting processes, including the use of ICT tools such as the LLP Link have not been solved yet (see the fourth Chapter of the case study) Effectiveness in reaching the target groups and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? According to the data collected for the case study analysis, the Erasmus Students Mobility for Studies action is successful in reaching both the institutional beneficiaries and the individual beneficiaries (students): - Vast majority of the eligible higher education institutions in Lithuania (45 in 2009) are awarded the University Charter. - The numbers of submitted applications by the individual beneficiaries significantly exceed the amount of funds. The dissemination of information about the programme to the students is organised via several sources: informational events, brochures, fairs, the Student association, the Erasmus student network, and informal student networks. Reaching the target group is considered effective given the fact that the numbers of student applications and inquiries had been constantly growing so far. 144

145 To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels 105 )? In case of the Erasmus Students Mobility for Studies the results of the action can be defined as: a) Positive experience of study abroad and the effects this experience had on the professional, academic or personal growth of an individual; b) The effects participation in the programme had on the internationalisation and the quality of the study programmes at the institution concerned, as well as on its management strategy. At the national level, the National Agency only implements measures of the outputs dissemination it disseminates, presents and publishes the general statistics on the student mobility. In case of other programmes, the results are disseminated during thematic dissemination events organised. For example, in 2008 the Quality competition 2008 covering all the LLP sectoral sub-programmes was organised in order to enhance valorisation and exploitation of the results of the programme. This year the NA organises an exceptional event of the Grundtvig town in Ukmergė town (related to the 10 th Grundtvig anniversary) with various creative publicity events and activities, good practice presentations. Also, the NA cooperates with the social partners (such as the Lithuanian Rectors Conference, Lithuanian Directors Conference, Lithuanian Student Organizations, higher education institutions, National Authorities, Chambers of Commerce, enterprises and other organizations) for ensuring wide dissemination of information on the possibilities offered by the programme (meetings and discussions with mentioned bodies are organised on various aspects of implementation of the programme). The major flaws related to the results dissemination and exploitation are these: firstly, despite detailed information collected about students mobility from the institutional reports, the NA does not perform any qualitative analysis on the results of the sub-programme and its impact on the national system of education and training. Secondly, no information on the sustainability or exploitation of project results is gathered after the contracts finish. The institutional beneficiaries themselves are responsible for the dissemination of the results at the individual level. There are no common national guidelines on how the results should be disseminated; hence each institution has come up with their own instruments. The most common solution is informal dissemination of the study abroad experience. The Erasmus coordinators often recommend to the students willing to study abroad to contact former Erasmus students with experience in the given country or study field. Another common measure is organisation of informational events where Erasmus students present their experience. One good practice example was identified at Vilnius University. A website was created where students describe their experience on the studying abroad, the living conditions and other practicalities about the host institution and country. Overall, in case of student mobility the personal dialogue is often the most effective way of informing the student and preparing him/her for the study abroad. The main finding is that despite the results in case of Erasmus SMS are less tangible compared with other programmes, the dissemination and exploitation of results are not sufficiently well organised. According to the National Agency, the highest possibility of the horizontal mainstreaming exists in case of the innovation transfer projects the results of these projects are often applied by other sectors or commercialised. For example, one of the private universities (ISM) develops and commercialises the academic study programmes based on the staff study visits results; one of the confectionary producers (SC Rūta ) has developed and introduced new products into the market as a direct result of the study visit to a factory in Belgium. Summing up, the data collected by the case study suggests that none of the stakeholders at the national level (neither the institutional beneficiaries, nor the National Agency or the national authorities) are sufficiently motivated to exploit and mainstream the results of the programme. None of the institutional beneficiaries interviewed saw a problem in the lack of exploitation of results. The national authorities were convinced that the beneficiaries should be more active in disseminating the results (both horizontally and vertically) and did not see their own role in this process. The National Agency on their turn was convinced that the national authorities should be more open to the results produced by the LLP and indicated a risk of duplication of 105 Horizontal mainstreaming refers to the project results and good practices adopted by other stakeholders. Vertical mainstreaming refers to the integration of the good practices and project results into general policy on national and/or European level. 145

146 efforts by the projects funded by the LLP and EU structural funds. Overall, it indicates the lack of national ownership over the results produced by the international EU programmes. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the Erasmus sub-programme that should be improved in order to increase the cost-effectiveness of interventions. Overall, available data indicate high efficiency of implemented action. However, there are gaps in the current management and control system, mostly related to the monitoring and reporting stage of the programme Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? In general the efforts to harmonise and simplify the Erasmus Students Mobility for studies action s management and control system are welcomed by the national stakeholders and there is an overall agreement that the situation is improving. The major complaints of relatively unstable management system and the constantly changing rules and administrative documents relate to the period. Furthermore, there are still many problematic areas, where the requirements for management, control or monitoring remain unclear or insufficient. Most of the problematic issues relate to the monitoring and reporting phases of the project cycle. The problematic areas are described below. The application stage. While project application process in case of Erasmus student mobility is considered as sufficiently clear and simple for both the National Agency and the institutional beneficiaries 106, there are small gaps that have a negative effect on the quality of the programme. Firstly, according to the interviewees, there are systemic problems with getting final application forms and other related documents on time. It would be beneficial if the final electronic application forms were provided at least two weeks prior to the application deadline and not modified afterwards. Secondly, some of the institutional beneficiaries (VGTU) were not aware of the final criteria according which the exact amount of funds allocated for funding the mobility grants for the institution is defined. They requested a clear explanation on how the funds are calculated and what is the role of the forecasted outgoing mobility in this process. Thirdly, smaller obstacles relate to: - Lack of the EU-level Erasmus students database where the institutions could easily check if the student had already received Erasmus mobility grant for studies (or placement). This database should be created at the EU level, because institutions at the national level do not have all data on the Lithuanian students that have completed part of their studies (e.g. bachelor or master degree) abroad. The National Agency possesses such database at the national level and it should be made available for higher educational institutions to use. - Different application deadlines at the host institutions create additional administrative load at the sending institution, and the even worse inconsistency in the cycle of determining the number of outgoing students and finding out the number of how many will be actually funded. The best solution would be to harmonise the application procedures not only at the EU level but also internationally at the level of institutions. The implementation stage. The programme implementation process in case of Erasmus Students Mobility for Studies action is relatively straightforward and clearly defined. The institutional beneficiaries are in general satisfied with the dialogue-based counselling approach at the National Agency and would not support the centralisation of the Erasmus programme. The major flaws relate to: - Firstly, the administrative workload due to the administration of students contracts, and the contracts between the institutions and their faculties. According to the institutional beneficiaries, a requirement to have a contract between the faculties is an unnecessary requirement that causes huge loads of paper work but little added value. This requirement also limits the choice of students. - Secondly, the delay of payments is a major problem for the institutional beneficiaries. The institutions often receive the Erasmus funds only after they have already signed the contracts with students or even send them abroad. Applications from the students are submitted in March, the deadlines for submitting documents to the host institutions are in April-June, but the sending institution will only 106 In case of the students mobility the opinion of the individual beneficiaries was not inquired. This gap will be filled by the data of individual project beneficiaries survey to be carried out by the contractor. 146

147 know the exact sum allocated for student mobility from the EC and the national budget in September. This inconsistency not only causes delays in making first payments to the students abroad, but also larger problems when the number of students forecasted diverge from the number actually funded by Erasmus (and this is usually the case). This type of problems when an institutional beneficiary has to return the remainder or search for additional funds decreases efficiency of the programme and the satisfaction of the individual beneficiaries. For increasing the efficiency, the decisions regarding the financial allocations of both national and EU funds to concrete institutions should be made no later than May. The reporting and monitoring stage. The information collected during the interview programme suggests that this phase creates most of the uncertainties or discontent of the national stakeholders. The discontent of the national agency and the institutional beneficiaries in general related to the same issues that are summarised in the text below. Firstly, efforts are duplicated when data have to be submitted both to the LLPLink and to the annual reports to the European Commission. The LLP link is not sufficiently used for monitoring purposes so far. The tool itself should be made more user-friendly (e.g. allow saving, printing the information, transferring data from Excel files). Moreover, the overall reporting cycle could be optimised if institutional beneficiaries were allowed to connect to LLPLink for downloading the monitoring information directly (this idea was also supported by the institutional beneficiaries). Secondly, confusion is caused by the amendments of the reporting forms (both for the NA and the beneficiaries) introduced not in a timely manner. More confusion is created when the final versions of the reporting forms are submitted by the EC very late before the submission deadline. This systematic flaw has not been corrected throughout the whole period. For example, institutional beneficiaries of Erasmus had to submit the final reports in November 2009, but the final amended report form was provided only in October. The form required additional monitoring information that was not in the contracts with students and lecturers. Hence the required information was not collected during the reporting period. According to the institutional beneficiaries, most institutions prepare large amount of monitoring information during the summer period when workload at the universities and colleges is lesser. Large universities that send abroad hundreds of students have loads of monitoring information on each student/lecturer to work on, so they hire apprentices to perform this work. Changes in the reporting forms after the monitoring information is ready cause the duplication of efforts and unexpected administrative load. Therefore it would be more efficient if reporting forms did not change after the contracts with students are signed. The necessary amendments can be implemented starting with the new period. Institutional beneficiaries also call for the reducing of the red tape of paper reporting forms and allowing electronic reporting (at least for submitting the monitoring data). Also, centralised informing of the beneficiaries about changes in the reporting forms or any other requirements should be organised in a more effective way. Currently the NA places a note on the changed forms on the website; most of the higher education institutions concerned do not notice this information. Information notice by would be a more effective solution. Thirdly, there is an imbalance between the need to have more qualitative evaluation of the results and impact of the LLP that reflects in the new monitoring requirements issued by the EC, and the capacities at the NA to perform this task. The NA complained about the lack of guidance or training provided by the EC (for example, in case of the system audits that the NAs had to perform in 2008). In addition, additional funding could be provided by the European Commission for the thematic monitoring activities, since the NAs do not have sufficient financial resources to pay for these activities. Summing up, the monitoring and control system of the Erasmus Students Mobility for Studies action focuses too much on financial supervision and control. Least attention is paid towards monitoring of results and on qualitative analysis of monitoring data. One can argue that analysis of results and ex post effects in case of students mobility are not that important, but lack of focus on results and impact is symptomatic in case of other projects funded by the LLP, such as the networks, partnerships, innovation transfer. The need for the more qualitative analysis of monitoring data in order to ensure its use in decision-making is hampered by the lack of capacity on behalf of the NA, and lack of interest from national authorities. The EC could invest into building the capacity by allocating additional resources to the NAs for thematic monitoring and analysis. 147

148 The unnecessary red tape might be reduced by: a) harmonising the administration procedures (documentation, financial rules and contracting) for all LLP decentralised actions; b) loosening some of the requirements for the inter-institutional communication (such as the necessity of bilateral faculty agreements) and the accountability of individual beneficiaries. This would help focusing on building the critical mass of the mobile students and increasing the quality of studies Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? In the reports of the NA there is a clear trend to complain about constant work overload. The administrative overload was also one of the main complaints of the institutional beneficiaries interviewed. The information collected during the case study analysis suggests that the administrative workload is caused by: a) Lack of competences (e.g. the analytical capacity) at the National Agency to perform certain functions that are newly introduced. - The NA mostly lacked capacity to perform qualitative research for monitoring/reporting purposes to the EC (e.g. when it is asked to evaluate the effect LLP had on the national education and training system, or the exploitation of the programme s results). - Another example is related to systemic audits at the institutional beneficiaries (a new requirement introduced in 2008). The National Agency used external assistance for conducting these audits. The external audit company was hired for providing guidelines and counselling on the conduction of 11 systemic audit visits to the institutional beneficiaries of the LLP. b) The unexpected changes in the administration and management requirements (see above). c) Numerous administrative and financial requirements related to the mobility of individuals as described further in this paragraph. The main source of red tape for the institutional beneficiaries is the management of the hundreds of contracts between the faculties and with the students, and ensuring that all the requirements (regarding studies transcripts, and other documents related to the studies and the financial accountancy of the mobility visit) are met. The details in the contracts, the content of study programmes and other specifics often change. Also, the work process cannot be optimised, as in each case it is based on a personal dialogue and counselling of the student. Personal dialogue is confirmed to be the most effective way of various problem-solving and increasing the satisfaction of the individual beneficiaries. The administrative workload could be diminished by reducing changes in the reporting requirements during the implementation of projects, and by allowing inserting some of the data directly into the database by the national agencies or the EC A lot of Monitoring information is being generated by beneficiaries. How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? Judging from the data collected for the case study analysis, most of the data collected through the monitoring reports of the institutional beneficiaries is never used for other purposes except reporting to the EC. Moreover, not all data, for example, concerning individual beneficiaries, are reported. Though the NA administrates a database containing data on each of the Erasmus student funded, this data is not generalised or analysed for any purposes, except for presentation of general statistics at various thematic informational events. 148

149 8. Erasmus Student Mobility for Studies (Incoming Students): Germany 1. INTRODUCTION The overall purpose of this case study is to provide an in depth insight (results based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme s action. The case study elaborates only on the incoming students mobility; therefore data were not collected on the outgoing mobility or the students placements in private companies. The complementary case study on the outgoing students mobility in Lithuania will contribute to the exploration of the full picture of the Erasmus students mobility for studies cycle. The information presented in this case study is based on the following sources: - Expert interviews with the national agency, the national authorities and the institutional beneficiaries of the Erasmus Student Mobility for Studies action. In five interviews a total of ten persons have been interviewed. Most of the experts (7 of 10) were interviewed face to face and the other by telephone (the detailed list of interviews is presented in the Annexes); - Desk research, which sought to analyse national documents, action s implementation statistics and other documentation, which was provided by the national agency, the national authorities and the institutional beneficiaries. Furthermore a series of results from other impact studies were incorporated into the analysis. The interviews and desk research were carried out during the period of 26 April 17 May RELEVANCE The main objective of this section is to describe the European added value of the selected action To what extent do the intermediate and specific objectives of the action remain pertinent to the national policy priorities? The primary objective of the Erasmus Students Mobility for Studies (SMS) action is to foster mobility of students in order to: 1) enable students to benefit with regard to education, linguistic capacities and culture from the experience of learning in other European countries; 2) promote cooperation between institutions and to enrich the educational environment of host institutions; 3) contribute to the development of a pool of well-qualified, open-minded and internationally experienced young people as future professionals in the national economy. It is not easy to outline national priorities in the German educational system at the national level because of the German federal system, from which the cultural sovereignty in education of the German federal states results. This means that the primary responsibility of legislation and administration in the areas of education, culture, etc., is assigned to the German federal states by the Basic Law of the Federal Republic of Germany. Nevertheless, there are general commonalities (similarities) and priorities which are represented by BMBF. In particular, the Federal Government supports institutions of higher education to adapt to new challenges arising from the internationalisation of higher education and increased competition. 107 As particularly relevant in the context of Erasmus appears the Initiative for Excellence, The Bologna Process and the Internationalisation of Higher Education: - The Excellence Initiative aims to advance similarly cutting-edge research and the quality of higher education and scientific fields in Germany at a broad range. The general goal is to strengthen Germany as scientific location, to promote its international competitiveness and to make top levels in the academic and scientific field visible. - An aim of the Bologna process is the introduction of a tiered system of Bachelor's and Master's degrees with comparable degrees throughout Europe, the introduction and improvement of quality management and growing together by increasing mobility in higher education. By doing this, on the

150 one hand the existing potential of knowledge can be better exploited and, on the other hand, higher education institutions can stay ahead in the international competition for the best brains. The Bologna Process started 1999, and - despite articulated criticism - has strongly contributed to the successful modernisation of German universities. - Agreements passed by the international bodies of the Bologna Process are implemented by the Federal Government and the German federal states in accordance with their respective areas of competence. The Bologna Centre of the German Rectors Conference is dedicated to a successful implementation of the Bologna reforms in Germany. It is financed by the BMBF and serves as a key advisory, coordinative and intermediary support unit for German universities. The activities of the Bologna Centre are covering the monitoring and dissemination of 'good practice'. - The Federal Government is supporting German universities to stand strong in the growing international competition. Attracting students, young scientists and researchers worldwide to study and research at German universities and to get top scientists and researchers from abroad to participate in scientific work at universities and research institutes in Germany is an important objective of the BMBF. The Sofja Kovalewskaja Award for excellently qualified young scientists, who will be able to work with their own working group for a three year period in a research area of their own choice, is only one example. Furthermore German institutions of higher education have been supported in developing their own study programmes abroad, as well as in enhancing their marketing at fairs und information events abroad. In this sense Erasmus is closely intertwined with national policy priorities and one of the most important precursors for these processes. Germany has set itself the target to increase the mobility of German students abroad and foreign students into Germany significantly. With regard to this Erasmus plays an important role in the European area. This assumption is also consistent with the priorities highlighted in the Leuven communiqué 108 that expressed the conviction that the mobility of students, young scientists and research personnel promotes the quality of the programmes and excellence in research as well as it strengthens the academic and cultural internationalisation of European higher education. Mobility is an important element of personal development and employability and fosters respect for diversity as well as the ability to deal with other cultures. To meet challenges of global competition, international exchange in education and training is given high priority. Thereby, outgoing students and incoming students are given importance to the same extent, because both in their own way contribute to an enrichment of teaching and research. The DAAD is an agency of the internationalisation of German higher education institutions and since 1987 has taken the role of a National Agency for Erasmus on behalf of the BMBF. In the academic year 2009/2010 (1 June 2009 to 30 September 2010), it will provide about 48 million for mobility and Intensive Programmes under Erasmus for 297 German higher education institutions (about 80% of German universities). In the opinion of national experts DAAD is the only instrument to produce coherence in the national priorities. In terms of internationalisation it tries to create something like a national education policy in great accordance with the EU objectives, which it is able to influence as well. With its expertise in this area the DAAD tries to do a lot to ensure coherence. The target of 20% of mobile higher education graduates in 2020, adopted by the Ministers in Leuven in 2009, corresponds to the level of Germany s national target. Furthermore the DAAD elaborated a National Action Plan to place Germany more favourable for incoming students. Therefore it is working with its own resources to improve the quantity of incoming students and has begun to develop a strategy that is described in more detail in section To what extent do the objectives of the action remain pertinent to the lifelong learning needs of the target groups? Experts from the organisations responsible for the incoming students express great satisfaction with the general conditions of the LLP and the conditions for their work set at the national level. They feel competently supported by the relevant institutions. The DAAD as a national agency is currently not responsible for the incoming students according to the 'sending country principle' (Entsendelandprinzip), but solely responsible for outgoing students, much to the regret of everyone involved. The concrete work with the incoming students will be provided mainly by the IO/AAAs and the LEIs, which are supported by the DAAD financially and conceptually. The IO/AAAs are mobility managers at the universities, in whose work Erasmus is one of the main elements. The fact that in the year of 376 higher education institutions 108 Kommuniqué der Konferenz der für die Hochschulen zuständigen europäischen Ministerinnen und Minister, Leuven/Louvain-la-Neuve, 28. und 29. April

151 nationwide had a valid EUC can be regarded as evidence of the universities strong interest in implementing the objectives of the LLP. The participation of the German universities can be described as excellent and as being much better than in other national programmes. 289 of 307 ERASMUS higher education institutions have made a mobility application for the academic year 2008/2009 to the NA. 288 higher education institutions have received a permit. The central task of the IO/AAAs is to take care of incoming students, the reception, registration and delivery to the study areas. In this sense they are responsible for the administrative care of the incoming students. LEIs consist primarily of local students who have recently returned from an Erasmus exchange and European students who currently study at a German host institution. LEI activities mainly concentrate on assistance and cultural aspects. They support the incoming students by orientating them in the new environment and helping them, for instance, with getting over timidity to speak German or organising social events. Stakeholders from IO/AAAs and LEIs regard the mobility of students as an important task and the incoming students as enriching for the life at the higher education institution and the daily life in their region. Therefore, they strive for further improvement of the local framework to increase the number of incoming students at their higher education institution and view it as great success if the incoming students decide to stay after their Erasmus-SMS. In order to promote the anticipated improved image, they want to have more access to adequate monitoring information to use it for further enhancement of their scope of services. This issue will be discussed more precisely later. By and large they have the impression of a general satisfaction of the incoming students with the results in the field of personal growth and professional development. This unfortunately cannot be proved by reliable data. Therefore, from the lack of criticism it can be concluded that a general satisfaction exists. A weakness of Erasmus is that it does not adequately reflect some of the recent developments (for example Bologna reform). The programme has to be modernised, and it has to be checked if the conditions for Erasmus grants (3-12 months, only partial studies) are still justified. An adjustment to the current changes in Europe must take place, including the question of the length of stays from shorter to entire study periods. Evidence showing the introduction of mobility accounts and the integration of additional programme elements for higher education will be given elsewhere The European added value. Would other national schemes / instruments provide enough support to student mobility related activities, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? According to the DAAD in recent years the education programmes of the EU have become important instruments for internationalising German higher education institutions. 109 One cannot imagine increasingly European and also international cooperation at the higher education institutions without these funding instruments provided by the EU. The European added value as a whole can be assessed substantially. In Germany, Erasmus is the main engine of mobility in Europe but affects overall only about one third of the entire international exchanges. Those are nationally supported, for example, by International Study and Training Partnerships (ISAP-Programme), Go East, bilateral exchange programmes, and Germanic Institute Partnerships (GIP), to name just a few. According to the experts, no other programmes could provide similar support nationally, which is due the complementarity of the national funding policy and European funding in Germany. Therefore, there is currently only a supplementary but no parallel national programme. It is assumed that a national programme in the same dimensions would not work, because Erasmus is based on reciprocity. A national programme would be incurred by much more expensive tuition fees and the effect would probably be lower. Without Erasmus many universities would question the effectiveness of their internationalisation efforts and consider the student exchange as 'useful but not necessary' in relation to the associated administrative work. Therefore the Europeanization of higher education without Erasmus is unimaginable, at small higher education institutions even more than at large ones. Also the institutions responsible for the assistance of incoming students fear that their existence without the existence of Erasmus would be called into question. Indeed the national promotion they get can be assessed as immense DAADs with national support are not so common in Europe but they regard their existence as closely linked to the programme itself. Without the support of the DAAD the activities for incoming students 109 DAAD (ed.), ERASMUS Annual Report

152 could only take place in a limited way because they do not raise enough money. Nevertheless, language courses and social events could be organised by the linguistic departments or the universities themselves. It is difficult to estimate in how far Germany could continue to maintain its attractiveness for SMS without additional money, being in competition with English-speaking countries and regarding the increased attractiveness of placements in the industry. The connection of Erasmus with national support policies and funding has to become even stronger in the future to allow an unlimited mobility in the European Higher Education Area. With regard to that, a coordination of the complementary national and European funding is of prior importance What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? Most of the interviewed experts would not say that there are any synergies resulting from the integration in the LLP. The links between the different strands are relatively low. The integration into the LLP brought no great disadvantage and no great profit. Organisationally the change was hardly noticeable for the stakeholder involved in the work with the incoming students, as they continue to act alone in the well-known element of the programme Erasmus. It is therefore assumed that some universities have not noticed any difference. The synergies at national events are limited to a common welcoming, followed by separate working groups. Links exist only in small areas, such as student projects in schools ('Europa macht Schule'). The following points of criticism could be identified: - In the whole, LLP is a very artificial roof. It should be examined whether it would be better to bring all the mobility schemes in higher education programmes (for example mobility schemes of Tempus and Marie Curie) under one roof, which are still currently separately distributed, and to build a large store where the higher education institution customer can help himself according to his needs. - The programme is considered overregulated, which will be touched upon again in section 4. - The national experts would welcome the pooling of SMS and SMP into a 24-month mobility account from which students can draw on demand. This would allow for a more flexible design of the mobility schemes adapted to individual needs and enable a more Europeanised design of study courses. 3. EFFECTIVENESS The main objective of this chapter is to provide suggestions on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The high level of participation of higher education institutions in Erasmus and the effects described in national reports and various studies are suggesting by and large a high effectiveness of the programme in Germany. On the institutional level incoming students and outgoing students trigger direct or indirect effects on the internationalization of higher education as well as efforts in enhancing the quality of the offered courses. However, due to the complexity of institutions and a non-continuous coverage of existing effects and satisfactions, only little reliable information on the effects of incoming students to the specific institution can be given. Rather reference is made to existing studies, which could elaborate that International Erasmus students on campus also have an effect on teaching methods and techniques as several universities noted that international student and staff exchanges bring different perspectives on teaching practices, often towards more learner-centred approaches. 110 Alone by their presence the incoming students have an impact on society and on the higher education institution, thus contributing to further development of intercultural learning, European awareness, the interest in languages and the European idea. It was consistently confirmed that connecting factors to a wide 110 European Commission, Directorate-General for Education and Culture (ed.), The Impact of ERASMUS on European Higher Education: Quality, Openness and Internationalisation, Executive Summary, December

153 range of possible effects are existing and that the associated programme s goals will be achieved on a small scale. The contribution of incoming students lies above all in areas outside of studying and teaching. As direct evidence of the incoming students effects on the higher education institution may be cited as follows: - ECTS as a very important tool for quality protection and transparency of all courses was introduced to the universities. With this higher education institution teachers do think about what they are actually teaching and mutual recognition of academic achievements can make progress. - With regard to the linguistic sphere the incoming students guarantee an important living contact with native speakers in seminars and lectures. - The incoming students are promoting and stimulating for the German students to be mobile themselves. They intensify and enrich the contacts of German higher education institutions to their foreign partners. - Single case approaches on the integration of incoming students sometimes lead to overall reform of modules. - Since Erasmus is based on reciprocity, the universities are trying to be good hosts for the incoming students with the hope that their outgoing students encounter similar conditions as well. Social events created in this context are an important focal point for permanent effects, as these mainly develop through personal contacts. Finally, basing on the 'sending country principle', higher education institutions rather focus on the outgoing students. From them they get a systematic feedback. According to the experts the incoming students are not very problematic and so suggestions for improvements are not easy to make What are the main external factors affecting the outputs and results of the action? On the basis of interviews and document analysis, four major external factors are identified that have a positive effect on the quantity of the Erasmus incoming students and their satisfaction with the outputs. These are the expected benefits of the stay in Germany, the appeal of the local environment, the attractiveness of the language of the host country and the general economic situation. The expected benefits of the incoming students stay in Germany are closely linked to the question, which area they feel is attractive. Germany, compared to Spain, does not have the so-called sunshine factor. It is rather regarded as country where it is suitable to make technical progress and to focus on academic achievement. Evidence can also be found if one draws a comparison to the SMP, where the popularity is higher than in SMS. This is due to the reputation of Germany as an industrial location. The attractiveness of the local environment is closely linked to the personal well-being of the incoming students and thus has indirect influence on the outputs of the stay. Satisfaction with the town, the friendliness of people, the climate, low price levels and the housing situation make an important contribution to the overall satisfaction with the stay abroad and this may have positive effects. Ultimately, it affects the extent to which incoming students, after their return to their home country, influence the willingness of other students to stay at the partner higher education institution. According to experts, the general economic situation in Europe has a detectable effect on the number of incoming students. This means, in times of economic boom retrogressive figures have been registered and in times of crisis they recovered again. It can be assumed that there is a connection with the general supply of employment, which is more available during periods of prosperity, and the possibility of a faster career entry as considered by the student. In the past a strong labour market has led many students to quickly graduate from university without studying abroad and achieving additional foreign country experiences. 111 However, if there is no opportunity for a quick career entry, the student can take his/her time for a stay abroad, and thus try to increase his/her job opportunities. The attractiveness of the language of the host country should not be underestimated. For a successful study in the host country it is necessary to deal with its language sufficiently. The basis for this is created at schools, where German as a foreign language is decreasingly taught, despite the fact, that German is the most widely spoken language in Europe. The less German is taught as a foreign language in European 111 DAAD (ed.), Erasmus Annual Report

154 schools, the fewer students are inclined to choose Germany for their stay abroad. Increasing mobility figures from Eastern European countries and declining or stagnating figures from Western European countries can be taken as an important hint on it. In the academic years 2006/07 and 2007/08 Germany was the top destination of almost all Eastern European Member States and Turkey. 112 There seems to be greater interest in the German language evolving from history. This issue will be discussed in more detail in chapter 3.2. No small part in the design and dissemination of these factors lies with the IO/AAAs, it is essential that they make positive factors more public and compensate for negative influences. Only by this weaknesses can be eliminated and new opportunities created Effectiveness in implementing horizontal policies How does the action contribute to the horizontal priority of multilingualism? The national experts fear a failure in the policy of multilingualism, because Europe is dominated by the English language. Earlier it was assumed that there is a concern of the German host institutions about the attractiveness of the German language. Behind this lies the effort to reach big numbers of incoming students, not just to advance the international nature of the institution but also to promote the diversity of languages within the institution. In order to increase the number of Erasmus incoming students, the DAAD developed a national action plan to significantly grow the numbers of incoming students from certain countries. A pilot project to improve the knowledge of the German language with free access for foreign Erasmus students to an online German language course has been developed in this context. Alone in 2009, the DAAD used 200,000 to offer these free online courses for incoming students which are tutored and for which they can get credits. The higher education institutions also make an effort to offer preparatory German courses to allow incoming students successful studies at their institution. They registered a decline of incoming students which have not enough German language competence. In their view this group of people is interested to study in Germany, but they cannot do so, because no courses in English language are offered. With regard to the implementation of the priority of multilingualism, English courses are a double-edged sword. On the one hand they do not promote multilingualism, and on the other hand they hinder the integration of the incoming students in the higher education institution and the local environment. In awareness of this politically sensitive issue, the development of a two-track strategy is being considered. English language modules would be one way to facilitate entry for the incoming students and ensure that they do not have to repeat the course content at their home university. German students would also be interested in such modules. At the same time courses in the host language should be compulsory to contribute to inclusion. The incoming students also need German language in daily life during their stay in the host country. A deficit is that there are no intensive language courses prior to the start of the stay; those do not exist for the German language. If one holds on to the goal of multilingualism, these intensive language courses must be extended to all official languages of the EU How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? Overall, it can be concluded that because of the 'sending country principle', host institutions have little or no effect on the compliance of the horizontal EO policy in selecting the incoming students, although Germany generally places high value on equality of opportunity. Moreover, higher education institutions are at the end of a chain of prior selection mechanisms that takes place already in primary and secondary education. Therefore the interviewed experts could not give a statement to this question in relation to the incoming students. However, it can be assumed that sufficient opportunities exist at German universities to ensure a successful guest stay, for example for people with disabilities and to provide assistance at the social services of the higher education institutions. 112 Statistics at < 154

155 3.3. Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? According to the national experts, the LLP has developed very satisfactorily concerning its possibilities, and it has a clear structure that can be demonstrated well to interested persons. Organisationally the change was hardly noticeable for the stakeholder involved in the work with the incoming students, as they continue to act alone in the well-known element of Erasmus. It is therefore assumed that some universities have not noticed any difference. Particularly welcomed are the opportunities for SMP, and that non-eu citizens studying in Germany can now participate in Erasmus too. This was a very big step in promoting to the international attractiveness of European educational space. Most of the experts interviewed would not say that there are synergies resulting from the integration in the LLP. The links between the different strands are relatively low. The integration into the LLP brought no great disadvantage and no great profit. The synergies at national events are limited to a common welcoming, followed by separate working groups. Links exist only in small areas, such as student projects in schools ('Europa macht Schule'). The following points of criticism were raised by the interviewees: - On the whole, the LLP is a very artificial roof. It should be examined whether it would be better to bring all the mobility schemes in higher education programmes (for example mobility schemes of Tempus and Marie Curie) under one roof, which are still currently separately distributed, and to build a large framework where the higher education institution customer can find a programme matching their needs. - The programme is considered overregulated, which will be touched upon again later. Prior to the LLP, it was different. With its introduction, the rules have been harmonised and therefore Erasmus has become administratively complex, and the external controls have grown enormously. The national experts would welcome the pooling of SMS and SMP into a 24-month mobility account from which students can draw on demand. Therefore it would be possible for students to study or have a placement twice or both. This would allow for a more flexible design of the mobilities adapted to individual needs and enable a more Europeanised design of study courses. Furthermore this idea could be implemented cost-neutrally Effectiveness in reaching the target groups and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? Overall, the number of incoming students in Germany is a major concern, and the ratio of incoming and outgoing students will be critically analysed. This concern is linked closely with the reciprocity-based basic principle of Erasmus. Since it is politically sought that as many German students as possible should study abroad, with an overall target of 20% of students, a disproportion between outgoing and incoming students should be avoided. To have more incoming students means indirectly to have the possibility to send more outgoing students at the same time. 155

156 Figure shows that the number of outgoing students in Germany in recent years has declined slightly, and the number of incoming students was stagnant. Concerning mobility balance, Germany is a net exporter. In the years 2006 to there are about more outgoing than incoming students. The mobility balance in 2007/08 of seven countries with the largest quantity of outgoing students is compared in Figure It clearly shows that it is not only Germany that has more outgoing then incoming students. In relation to Poland, the disproportion is relatively low. But measured by the disproportion level, Germany ranks second amongst these countries. Yet in recent years, Germany was always the host country number 3 in the European total balance, behind only Spain and France. At the same time, most incoming students to Germany come from France and Spain. 113 No figures are available for incoming students in 2008/

157 Figure German mobility figures by year (number) total incomings outgoings /05 05/06 06/07 07/08 08/09 year Sources: Figure Proportion of outgoing and incoming students in 2007/08 by country. 3,50 3,00 2,50 2,00 1,50 proportion 1,00 0,50 0,00 DE ES FR IT PL UK PT Source: 157

158 Figure Top 7 German incoming students in 2007/08 compared with outgoing students by country (number) total incomins outgoings FR ES PL IT CZ UK HU country Source: Figure also signifies another basic problem in mobility. Looking at individual countries, the balance of exchange varies significantly. This is particularly the case in Eastern European countries, for which Germany tops popularity charts, but they rank much lower as a destination for German students. The interviewed experts emphasise two significant issues: a) The National Action Plan which should increase the number of incoming students and contribute to a significant improvement of the exchange balance; b) The question of the influence of the Bologna process on the overall mobility of students. What conclusions can be drawn from the falling or stagnating mobility figures derived in this regard? As already mentioned above, the DAAD elaborated a National Action Plan to increase the numbers of Erasmus incoming students from certain countries significantly. Germany will be better placed for incoming students, although it has already achieved a high level. Therefore the DAAD is working with its own resources to improve the quantity of incoming students and has begun to develop a strategy that consists of several components: - An example was an event made together with the British agency for British universities to influence the multipliers, since the UK is a very attractive destination also for German students. - A pilot project was elaborated to improve the knowledge of the German language. It provides foreign Erasmus students free access to an online language course to ensure that the incoming students can study more efficiently abroad (see 3.2.1). - Bilateral seminars with countries, with which improvement of exchange is targeted, are organised. The contact seminars were started with all of the Central and Eastern European countries and will be extended to the Western countries. The background for this is that the exchange of students is influenced by multipliers (higher education institution teachers). It is necessary to convince the multipliers. If the teachers are not convinced they will not convince their students. Therefore, meetings between professors from different countries have been organised with the aim of ensuring that agreements will be made as a result. - Another measure, the LEIs, are designed to improve service and integration of the incoming students. This work is done by the IO/AAAs as well, but 'peer to peer' it is more the accepted than from the official position. They are promoted by the DAAD and often consist of former Erasmus students. This clarifies that Germany develops a set of measures to improve the exchange balance. There are national funds from the BMBF available, to which the theme of integration and attraction of foreign students is important. More work has to be done with regard to the visibility of these efforts. Although in interviews national experts mention influences of the Bologna process on the mobility, it seems too early to draw definite conclusions. The extent to which the decline of studies abroad is attributable to the introduction of the compact three-year bachelor's programmes has to be assessed in particular later. It is also possible that difficulties in the beginning phase of the new LLP are responsible for that (at the start of 158

159 Socrates/Erasmus 1996/1997 it was similar). Statements of the affected countries suggest that other factors also can be responsible for the weakening development of mobility in Europe (for example a good labour market, introduction of tuition fees, increased demand for a stay outside Europe). However, one effect of the Bologna reform seems certain: the universities have to pay more attention to the curricular framework of mobility and the question of recognition of studies abroad, so that the target of boundless mobility in the European Higher Education Area can be achieved. With regard to the question of study funds the recognition of study credits earned abroad gets increasingly important. Students whose credits are not accredited at all or only accredited partially have to study longer and thus have additional costs. Moreover, in the conception of courses the transparency and compatibility of the curriculum, mobility windows and the development of mobility-friendly curricula have often been neglected. The students in their course of studies need a well-defined space where phases of mobility can be inserted. Additionally in Germany there are very short BA courses (three years) which are associated with time pressure. In the context of the accreditation procedure of the higher education institutions mobility windows have to be accounted for. While the concept of mobility windows has just been characterised with the Declaration of Leuven 114, the joint communiqué suggests the importance of this measure to keep up student mobility in Europe. Many higher education institution teachers still do not take the recent developments with regard to ECTS seriously enough. The higher education institution administration has to be mobilised to complement their effort. With regard to the national and EU-wide target of 20% mobility the main question is, whether the stagnation at a high level of SMS gives possible evidence of a degree of saturation To what extent are the programme results adequately disseminated and exploited (at both horizontal and vertical levels)? With regard to the Erasmus SMS the results of the action can be defined as: a) The positive experience of study abroad and the effects this experience made on the professional, academic or personal development of an individual; b) The effects made on the internationalisation and the quality of the study programmes at the higher education institution, as well as on its management strategy. By documentations of good examples discussed at information events at the universities, the DAAD tries to show how stable and sustainable links among universities in Europe (particularly in less popular target regions) can be established. Statistical analysis and examples of good practice will also be used to detect the effect of the newly introduced BA/MA programmes for mobility and to clarify the question of how mobility can be secured and sustained in the Bologna structures. 10 Erasmus experts from the higher education institutions supporting the work of the DAAD act as speakers in events and explain their programme experience to the participants as informants by telephone and . The IO/AAAs and LEIs, which foster the dissemination on-site, play an important role in this context. A regular exchange of experience is organised by the DAAD. Each year the DAAD prepares Erasmus annual reports 115 presenting achieved results and keeping record of the results of the programme s work. Welcome notes by the BMBF are highlighting the enormous importance of the programme and the work provided by the DAAD. Statistical analysis is significantly limited to the presentation of the total outgoing students and contains less information about the incoming students. Higher education institutions use incoming students in special events (country evenings) to provide information about their country, city, university, and to advertise mobility. Other examples of 'good practice' of dissemination are print products 116, in which Erasmus students tell about the experiences gained during their residency in the host country. Although they focus mainly on outgoing students, a small number of reports written by incoming students are included as well. Being part of the nationwide project "Europa macht Schule", European exchange students attend classes in a local school or conduct a project in which they present their home country in a creative way. The aim of this is to introduce pupils to incoming students 114 Kommuniqué der Konferenz der für die Hochschulen zuständigen europäischen Ministerinnen und Minister, Leuven/Louvain-la-Neuve, 28. und 29. April e.g. DAAD (ed.), ERASMUS Annual Report 2009, DAAD (ed.), ERASMUS Annual Report

160 at an early stage and to motivate them to study abroad. 117 Articles in newspapers, in which incoming students give information about themselves in their mother tongue, are another example. To sum up, several steps to make incoming students visible are taken by the higher education institutions and the DAAD. The social events organised by the LEIs and IO/AAAs provide a further contribution. One is aware of the fact that on the incoming side not all dissemination possibilities have been exhausted yet and that an exchange of good practice is necessary and worthwhile also in the future. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the sub-programme that should be improved in order to increase the costeffectiveness of interventions. Overall, it is estimated by the interviewed experts that the overall development of the LLP and its implementation has been very positive. The administrative effort by incoming students is comparatively negligible. However, some gaps and suggestions for improvement are identified Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? In the coming years, the DAAD will strongly promote further expansion of the EU programmes and argue for a substantial simplification of programme management. This has become more difficult again in recent years, particularly in view of Erasmus, despite all expressions of political willingness. Reasons for this are the integration in the LLP. Previously, all were satisfied with the programme management. With the introduction of LLP, the rules of the sub-programmes have been harmonised and therefore Erasmus had to adapt stricter rules of other sub-programmes; external controls have grown, and the reporting requirements are more complicated. The national experts advocate to focus more on an 'order of proportionality' and not to stick to an 'order of equal treatment'. This and a simplification should be a main focus in the development of the next programme generation. In addition to de-bureaucratisation of the administration, there is a need to reform the 'past-performanceprinciple' towards a focus on the real numbers. This would increase the financial incentives for SMS and allow greater mobility figures Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are the key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? Although the DAAD spends millions of national funds provided by the BMBF in the IO/AAA s for the support of incoming students 118, without which the universities would be reluctant to act in this area, the studyrelated services in Germany are estimated as generally underserved. A reason for this is the setting of priorities at the local level, which is connected to the budgeting of higher education (higher education institution funding). The share of funding for international activities is therefore no longer set politically but by the balance of power in universities. The number of German outgoing students is slightly declining, and higher education institutions have less inclination to deal with the complex management of the programme. In addition, the universities increasingly have to think and act entrepreneurially and evaluate through costs and benefits of activities more closely. 119 The national experts mention that in a series of higher education institution's the IO/AAA s are underequipped in terms of human and financial resources, which prevents a lot of additional activities. The importance of their work has to less been realised at the local level. At this point, the DAAD gives only basic support in the form of provided overheads, and the universities come up with the rest on their own. One The agencies in other countries are viewed to be less capable in terms of resources. 119 "Overall respondents mentioned that the bureaucratic costs in terms of time and administrative procedures required during the tendering and participating in centralised actions often are relatively high compared to the benefits of the projects. About one fifth of the institutions consider Erasmus to be costly and absorbing too many administrative, financial and human resources." (European Commission, Directorate- General for Education and Culture (ed.), The Impact of Erasmus on European Higher Education: Quality, Openness and Internationalisation, Executive Summary, December 2008) 160

161 solution would be to calculate the overhead not only by the outgoing students but also take the incoming students into account, because they demand administrative work as well. It could create more economic incentives for renewed commitment A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? National agencies abroad and the EC itself collect a lot of monitoring information on the mobility. National experts bemoan that they have, for example, no access to the Erasmus final reports of their incoming students, which are prepared as an account for the sending country. These data could improve efforts for the attraction and support of their incoming students significantly. However, the DAAD and the higher education institutions have a number of activities to better use monitoring information. - In a project financed by the BMBF, the DAAD handled the issue of transnational mobility in Bachelor and Master programmes. As part of the project, results of a survey of 14 Bologna countries and an international conference on transnational mobility have been integrated into a manual In a statistical volume the DAAD has the quantitative results of the various Erasmus activities documented, analysed and evaluated. The data provide information about which regions, universities, subjects and target countries gained particularly good or less satisfactory results. This analysis is now used, for example, by some German federal states to enhance the performance of their higher education institutions Through visits to a selected number of universities the DAAD carried out together with Erasmus and Bologna experts, compliance with the EUC, the importance of Erasmus for the internationalisation of higher education policy, activities to improve the quality of mobility (for example accreditation) and problems in implementing the Programme, were discussed with higher education institution administrations, representatives of faculty and students, and possible solutions were debated. Examples of good practice were identified during these visits as well Some universities collect and analyse the satisfaction of the incoming students on evaluation forms. The mobility of the teaching staff is also used to talk directly to the partners about grievances and improvements. - Exemplary, the DAAD has analysed the reports of the British outgoing students to find out something about the German reputation with reference to the National Action Plan to increase the number of incoming students. Anonymous expert describes a system error of Erasmus that host institutions do not receive the reports of the incoming students. From the reports, they hope to draw conclusions to increase the number of incoming students. Although the student reports are relatively standardised, there is no database where they can be accessed by the host higher education institutions. It is suggested to set up a Europe-wide database into which all student reports are entered, and the analysis could be used for the successful development of the programme itself. 120 DAAD (ed.), ERASMUS Annual Report National Agency Report National Agency Report

162 9. Erasmus Staff Teaching Abroad : the Czech Republic 1. INTRODUCTION The current LLP has been running for four years. In order to improve the continuation of the Programme and to provide recommendations for the successor programme ( ) the EC/Directorate-General for Education and Culture has launched this interim evaluation. The objective of a retrospective analysis is to evaluate the programme implementation and its results. The prospective analysis intends to examine strengths and weaknesses of the programme in order to provide recommendations for its continuation as well as for the preparation of a successor programme. An overall purpose of this specific case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including costeffectiveness) of the evaluated Lifelong Learning Programme s action Czech Staff Teaching Abroad. The case study elaborates only on the Erasmus activity related to the Czech outgoing staff teaching abroad mobility; therefore data were not collected on students, the incoming staff mobility or the staff training abroad. The information presented in this case study is based on the following sources: - In-depth interviews with the national agency, the national authorities and the institutional beneficiaries of the Erasmus Staff Mobility. In total 10 persons have been interviewed (the full list of interviewees is presented in the Annexes); - Desk research, which sought to analyse national strategic documents, action s implementation statistics and other documentation, which was provided by the national agency, the national authorities, the institutional beneficiaries and social partners. The interviews and desk research were carried out during the period of 7 May 20 June RELEVANCE The main objective of this section is to describe the European added value of the selected action To what extent the intermediate and specific objectives of the STA remain pertinent to the national policy priorities? Erasmus Staff Mobility (STA) - Teaching assignments by higher education institutions teaching staff has the following objectives: - To encourage higher education institutions to broaden and enrich the range and content of courses they offer; - To promote exchange of expertise and experience on pedagogical methods; - To create links between higher education institutions; - To allow students who do not have the possibility to participate in a mobility scheme, to benefit from the knowledge and expertise of academic staff from higher education institutions in other European countries; - To motivate students and staff to become mobile and to assist them in preparing a mobility period. These objectives are highly pertinent to the national policy priorities. There is a high level of coherency between the objectives of Erasmus STA and the long-term objectives of the national higher education policy in the Czech Republic. 162

163 The priorities of the Long-Term Plan of the MoEYS123 are focused on three major areas: - internationalisation, - quality and excellence of academic activities, - quality and culture of academic life. The implementation of the priorities of the Long-Term Plan of the Ministry of Education, Youth and Sports is supported by providing the necessary co-funding. The total sum for Erasmus programme in Czech Republic in 2008 was This constitutes only 21,5% of all resources dedicated to this programme. The sum of came from the complementary national co-funding from the state budget (MoEYS), that forms 88,5% of the resources. The way how MoEYS supports STA are the developmental projects of higher education institutions (under the budget line Development Programmes ). The aim of Development Programmes is to contribute to the achievement of the priorities set out in the long-term strategy of the Ministry and its updates To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? The Strategy of Lifelong Learning in Czech Republic (MoEYS, 2007) approved by the Government and is regularly updated does not forget university teachers and guides to in the framework of development of the pedagogical and research qualities of academic staff, emphatically support their international mobility. 124 Different target groups benefit from the support of the Erasmus STA action: teachers (as direct individual beneficiaries), the higher education institutions awarded the European University Charter (as indirect institutional beneficiaries), and students. The Erasmus STA Mobility action remains highly pertinent to the needs of all target groups. STA promotes exchange of expertise and experience on pedagogical methods. Teachers get a chance to examine new tuition systems, get experience on different student attitude towards learning and on new teaching methods. Teacher mobility results in the more open style of teaching and willingness to innovate the teaching systems at home institutions. Teachers increase their language skills and make new professional contacts. The main benefits of taking part in the action for teachers relate to: - Increased knowledge of new teaching methods, possibility to compare tuition systems and curricula; - Professional development of teachers, increase of language skills (e.g. more publications in foreign languages); - Contribution to the development of mutual cooperation between the partner institutions (e.g. joint study programmes), increase of student exchanges. What concerns the benefits for students, Erasmus STA action allows students who are not able to participate in the Erasmus mobility scheme to benefit from the knowledge and expertise of academic staff from higher education institutions in other European countries. Czech institutions try to increase the range of courses which are partly provided by hosting teachers. Teachers with Erasmus experience try to introduce new courses taught in foreign languages at their home institutions. 123 The Long-Term Plan for Educational, Scientific, Research, Development, Artistic and Other Creative Activities of Higher Education Institutions for ( in English) and The Long-Term Plan for Educational, Scientific, Research, Development, Artistic and Other Creative Activities of Higher Education Institutions for ( in Czech) 124 The Strategy of Lifelong Learning in the CR (MoEYS, 2007, 163

164 The main benefits for students are: - Possibility to attend courses taught in foreign languages - new teaching methods, increase of language skills, and contact with international students; - STA often combines more activities - e.g. preparation of other types of Erasmus cooperation activities, promotion of home institution etc. and results in increased student exchange (more outgoing and incoming students) What is the European added value? Would other national schemes / instruments provide enough support to related activities, if there was no LLP? According the data collected and opinions expressed by the interviewees, the European added value by the programme is high. If there was no LLP, the international mobility would still exist but in much more limited scale. The reasons are demonstrated in the following data. If comparing total numbers of teachers in the country and the numbers of teachers on Erasmus teaching assignments, the Czech Republic has one of the highest ratios of outgoing Erasmus teachers in Europe. 125 When institutions are compared by numbers of teaching assignment, Charles University in Prague was ranking 1st in the last statistics available (2007/2008) 126. It is not a chance; in the two previous academic years there were always three Czech universities among the six institutions with the highest outgoing Erasmus teacher mobility. 127 Erasmus is the most popular programme among LLP programmes and the most widely used among public universities. 22% of all international mobility visits of the staff of public universities happen due to the existence of Erasmus programme. The following table demonstrates use of the different LLPs for teachers mobility and the fact that Erasmus STA is far the most popular programme. Table Public universities academic staff international mobility in Czech Republic in Programme A.1 Erasmus A.2 Comenius 52 A.3 Grundtvig 23 A.4 Lingua 14 A.5 Minerva 0 A.6 Leonardo 45 A.7 Jean Monet 0 A.8 Erasmus Mundus 15 A. Sum of LLP B.1 CEEPUS 101 B.2 AKTION 17 B.3 Other 285 B. Sum of other international mobility besides LLP 403 C.1 Government scholarship 22 C.2Direct interuniversity cooperation in Europe 569 C.3 Direct interuniversity cooperation out of Europe 75 C. Sum of government and universities programmes 666 SUM TOTAL Source: Annual Universities Reports, CHES 2009 Mobility 125 Outgoing and incoming Erasmus staff mobility for teaching assignments in 2007/ See and 164

165 2.4. What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? No obvious cases of duplication of efforts were identified. On the contrary, the organisational arrangement allows good cooperation and synergies among different mobility programmes. The preconditions for the coherency of efforts at the national level were created by establishing a separate agency for coordination of the LLP. Common actions and activities for different programmes are now coordinated. Maintenance of various forms of international mobility is in Czech Republic concentrated in the House of International Services MoEYS (DZS, which is an organisation established by the Ministry of Education, Youth and Sports for the one special purpose. Since 2007 the House of International Services MoEYS includes also the National Agency for European Educational Programmes (NAEP, incurred in connection with the new generation LLP and in which the activities of the National Agencies for Socrates and Leonardo da Vinci were transformed. NAEP is responsible for the administration of LLP and its sub-programs (including Erasmus), and administers other international programs: Erasmus Mundus, Tempus, FM EHP/ Norway, SCIEX-NMSch Schwitzerland, etwinning. NAEP provides promotion of educational programmes, assistance and consultancy services, management of funds and contracts for decentralised actions, monitoring, dissemination and valorisation, organisation of seminars and conferences. The House of International Services MoEYS further ensures international programs AKTION Czech Republic Austria, CEEPUS and programs of MoEYS - Countrymen, Lectorates, Calibrate and Studies of Foreigners in the CR. Cooperation with the Fulbright Commission in the CR is functional. DZS is also part of the Academic Information Agency (AIA) is another part of the House of International Services MoEYS. It provides information services to both students and university teachers, but also provides competitions for scholarships abroad under international agreements and others. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The overall effectiveness in meeting the targets and objectives of the action is high. Some basic information on Czech academic international mobility in general and Erasmus STA in particular illustrates the general picture. The numbers of higher education institutions in the CR are: 26 public universities, 2 State universities and 45 private universities. The staff of public universities is formed by full-time jobs, from it are academics. There were registered higher education students by 1 January According to the annual universities reports, there were international visits with only registered in formal programmes. The high majority of the visits is thus organised in other frameworks (research grants, projects, direct arrangements among university departments and concrete academics etc.) for variety of reasons. 165

166 The average number is 0,49 visits per academic of a public university (half of them travel abroad for an official trip every year). There is a high variability among universities: from 1,49 on one site (VŠCHT) to 0,016 on the other (VFU; only from sixty travels). As there are equal opportunities to use the funding available, the differences in the usage of mobility depend on the management and administration of various universities. The use of LLP for international mobility in one year varies between 38,4% of academic staff (Ostrava University) and 1,7% (Veterinary and Pharmaceutical University). The total ratio is 11,9%. One of the reasons for high participation of the Czech teachers in Erasmus STA programme could be the fact that the senior teachers were not allowed to travel abroad during communist regime and they are utilising the new opportunities with enthusiasm. The second reason could be the good organisation and PR of the LLP both on national and universities levels. Beneficiaries are highly satisfied with the outcomes of the Erasmus STA programme. Every year the numbers of university staff using Erasmus programme increase. The numbers of staff mobility for teaching assignments has grown by six times in ten years. 128 The growth was not stable but fluctuating. The intention of NA is to increase the quality and keep the current volume of staff mobility. The growth of the numbers of outgoing teachers should not exceed 2% in What are the main external factors affecting the outputs and results of the action? The main factors that positively affect the outputs and results of the STA mobility relate to the level of development of both national economy (the overall social and economic situation and the standards of living) and to the level of development and competitiveness of the national education and training system. The standards of living and the economic and social development level in the Czech Republic are permanently growing. Support of teachers international mobility is a part of long-term national education strategy. The complementary national co-funding from the state budget (MoEYS) forms 88,5% of the needed resources. Concerning negative external factors the interviewees mentioned only unexpected and unpredictable events (such as the volcanic eruption in Iceland). All the other issues are manageable Effectiveness in implementing horizontal policies How does the action contribute to the horizontal priority of multilingualism? Overall, it may be concluded that the Erasmus STA action contributes directly to the multilingualism objective by improving the language skills of its participants. The numbers of outgoing teachers are slowly increasing to most destination countries every year, but mainly to the new Member States (ES-10) of the EU. The favourite countries for teacher mobility are Germany (15%), Slovakia (13%), Poland (9%), France (8%), Spain (7%) and Great Britain (6%). Erasmus helps to achieve the objectives of the Bologna Process (promoting of mobility, quality assurance, European dimension etc.). Teachers with Erasmus experience often try to introduce new courses taught in foreign languages at their home institutions. This action is thus an important instrument for internationalisation a key priority of Czech higher education policy. 128 ERASMUS CR Staff Teaching Abroad Statistics. 166

167 How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? The issues of equal opportunities are implemented on national level; national rules correspond to the EU rules and EO issues are addressed during trainings. As the higher education institutions are autonomous and decide independently, their approach to EO could vary. Concerning age of the outgoing teachers, the present division is the following: - junior (29%); - intermediate (45%); - senior (26%). The serious issue is that 25% of outgoing teachers in the academic year 2007/2008 participated in the Erasmus teacher mobility for the first time, which means that 75% of participants use Erasmus resources repeatedly. More supporting approach should be taken towards implementing the equal rights of women and young teachers with no previous Erasmus STA experience. But it is possible only indirectly, no quota approach is possible in the framework of universities autonomy Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? There is sufficient evidence that the integration of the LLP on national level was successfully implemented. The National Agency for European Educational Programmes (NAEP) was established by the Czech Ministry of Education in 2007 as a part of Centre for International Services MoEYS. NAEP is a successor of the Socrates and Leonardo da Vinci National Agencies and is responsible for implementation of the Lifelong Learning Programme and other educational programmes in the Czech Republic. The advantages of the integration of the previous activities into the Lifelong Learning Programme are obvious: the numbers of participants are permanently increasing. The beneficiaries appreciated simplification of the administration after the integration. There are synergies related to the more systematic approach towards the administration of the LLP, mostly related to the information provision and reporting. Integrating the previous programmes into one LLP has allowed better reach of the target groups and more efficient use of the administration resources at the NA Effectiveness in reaching the target groups and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? Erasmus STA is successful in reaching both the institutional beneficiaries and the individual beneficiaries. Vast majority of the eligible higher education institutions in Czech Republic are awarded the Erasmus University Charter. The number of higher education institutions is permanently growing from 42 in 2006/2007 to 68 in 2009/2010. If compared with student mobility, staff mobility for teaching assignments is more balanced across all the subject areas. Subject areas, which show under-representation in student mobility (e.g. education/teacher training, natural sciences), are relatively well represented in staff mobility for teaching assignments. 167

168 To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? The Programme results are adequately disseminated at the national level; there are still reserves left at the level of higher education institutions. The dissemination of information about the programme at the national level is organised by NAEP by various means: monitoring meetings with Erasmus coordinators, dissemination seminars, valorisation conferences, website, Facebook group on Erasmus in the Czech Republic, publications, and LLP Info days in different regions of Czech Republic. The institutional coordinators themselves are responsible for the dissemination of the results at the level of their institution. There are no common national guidelines on how the results should be disseminated; hence each institution has come up with their own instruments and own results. Teacher mobility results in more open style of teaching and willingness to innovate teaching systems at home institutions, students of home institutions of Erasmus teachers can benefit from their new experience and expertise. The most common way is the informal dissemination of the personal experience. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the costeffectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? Overall, judging from the data collected, the present legal and institutional framework of programme management is optimal, generally accepted by the National authority, all interviewees in NAEP and the beneficiaries too. Synergy was achieved among different programmes that have an impact on other funding schemes and activities. At the national level the workload is high, but persons in the NAEP are rather enthusiastic. The situation differs in different higher education institutions. According to the statistical data, participation of some higher education institutions in the Erasmus STA is not as high as it could be, but it could be also a personal factor. The interviewed beneficiaries are very satisfied and pleased by high numbers of teachers they helped to travel Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? Under the present circumstances the present situation is optimal, the available resources are adequate. Some companies are used for external audit (Vox Consulting financial part) and internal audit (provided by external company Forward Line). 168

169 4.3. A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? The use of monitoring information is not intensive. Most of the data collected through the monitoring reports of the institutional beneficiaries is used only for purposes of incorporation into the NA reports to the European Commission. The reports generated by the beneficiaries are available, but not exploited. 169

170 10. Erasmus Staff Teaching Abroad : Norway 1. INTRODUCTION The overall objective of the case study is to investigate the functioning of the LLP Erasmus sub-programme, in particular the Erasmus Teaching Abroad (outgoing teachers) action in Norway. Norway was selected as the country for this case study, as statistics showed that Norway has together with nine other countries relatively low numbers of outgoing teachers. The other countries are Germany, Greece, Ireland, Italy, the Netherlands, Spain, Sweden, Turkey and the United Kingdom. In addition, fewer teachers were involved in Norway in (325) compared to (345). These facts led to the description of Norway as a reluctant country concerning Erasmus Teaching Abroad in the Inception Report. 129 At the same time it is relevant to notice that the number of incoming teachers (to Norway) on Erasmus assignments is growing: from 321 in to 367 in The figures for are not available yet. The national agency in Norway (SIU) receives the figures for the incoming teachers from the European Commission, has no detailed information about these teachers and does not know at which organisations the incoming teachers stay in Norway. Therefore it was decided to concentrate in this case study mainly on outgoing teachers. The case study focuses on three main factors: relevance, effectiveness and efficiency. The information presented is based on: - In-depth interviews and exchange with representatives of the national agency, the responsible Ministry of Education and Research, institutional beneficiaries and members of the national Erasmus programme committee; - Desk research of documents the NA and the institutional beneficiaries or downloaded from websites. The desk research and interviews were carried out in the period of 22 April and 28 May RELEVANCE 2.9. To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the national policy priorities? What are the links between the LLP and national policies? The specific objectives of the Erasmus Teaching Abroad action are to support teachers to become involved in international exchange of expertise and experience on pedagogical methods, to encourage higher education institutions to broaden and further develop their curricula, to create links between higher education institutions and enterprises and to support mobility actions from other staff members and students. These objectives are highly relevant to Norwegian national policy priorities, which was underlined by the interviewee at the Ministry of Education and Research. Similar objectives are described in the Norwegian White Paper Internationalisation of Education in Norway 130 : Universities and university colleges should provide higher education at a high international level 131 and Internationalisation of higher education should lead to attractive and relevant curricula and should improve the role of higher education institutions in society. This process should be driven by the demand for equal educational opportunities for everyone. 132 The White Paper clearly encourages internationalisation. In the case of teacher mobility it states that higher education institutions should be aware of the language competence of their teachers. The White Paper includes no concrete incentives for promoting teacher mobility and it refers to the responsibility of each higher education institution to work towards Internationalisation (see 3.1.1). The White Paper corresponds to Norway s active involvement in the Bologna process and followed the Quality Reform of The Quality Reform contained the implementation of new degree systems (at bachelor, masters and PhD level), the introduction of the European Credit Transfer System (ECTS) and new grades (A-F) as well as a number of changes to administration and management within the higher education 129 European Commission, Directorate-General for Education and Culture (2010), Service Contract No PPMI, Interim Evaluation of the Lifelong Learning Programme , Final Inception Report, page Norwegian Ministry of Education and Research (2009), Internationalisation of Education in Norway Summary in English: Report No. 14 ( ) to the Storting and Det Kongelige Kunnskapsdepartement, St.meld.nr. 14 ( ) Internasjonalisering av utdanning, Oslo. The Storting is the Norwegian Parliament. 131 Described in the Norwegian Act relating to universities and university colleges (2005), see: and (in Norwegian). Norwegian Ministry of Education and Research (2009), page

171 sector. The latter contains the establishment of the national agency SIU and the Norwegian Agency for Quality Assurance in Education (NOKUT) To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the lifelong learning needs of the target groups of the programme? Even though the objectives of Erasmus Teaching Abroad actions are highly relevant to Norwegian national policies, the number of teachers going abroad declined from to (from 345 to 325 individual beneficiaries), which could be seen as indicating that the objectives were less relevant for the teachers and their employers. This judgement is not correct. All interviewees confirmed that the objectives are relevant. It is important to note that the number of outgoing teachers had been increasing before (from 296 in to 345 in ) as had the number of incoming teachers (from 321 in to 367 in ). The decrease in outgoing teachers in can be attributed to several factors: a) Teacher mobility has no high priority, either at European or at national and institutional level. Even though the NA confirms that teacher mobility is important, the NA can only finance a certain number of mobility grants for teachers; the main focus of the EC and national authorities is on financing mobility grants for students. 133 This is also mirrored in the fact that little is documented concerning Erasmus Teaching Abroad; recent national evaluations are not available. 134 The main focus in the above mentioned Norwegian White Paper is on student mobility. b) Very often the institutional beneficiaries overestimate the need for mobility grants in their applications (they apply for more grants than they use), which makes it difficult for the NA to handle the budget available for Teaching Abroad grants. The interim reports which the institutional beneficiaries send to the NA and in which they report on the use of the mobility grants are very useful. The NA can adjust the number of grants after having received this information. Institutional beneficiaries overestimation of their need for mobility grants is caused by the fact that the interest in teaching abroad is higher than the implemented cases: sometimes teachers cannot travel because of professional (e.g. teaching obligations, see ) or private reasons. How often this is the case, it is not evaluated. c) Teachers at universities are less interested in Erasmus Teaching Abroad grants as they are more interested in research and research exchange; therefore they are more attracted by the European Research Framework Programmes, which does not mean that they do not share the objectives of the LLP. In addition the universities main focus is on mobility grants for students. It is necessary to provide and improve information for teachers at universities about Erasmus Teaching Abroad grants. Holding the Erasmus University Charter is one pre-condition for using Erasmus and shows that organisations in Norway are interested in international exchange: 55 organisations hold an Erasmus University Charter, but not all make use of Erasmus Teaching grants and very often the same teachers use mobility grants every year or every second year. Organisations which have a well developed and clearly articulated international strategy are more active Published SIU reports on mobility grants in 2009 contain mainly information about student mobility: SIU (2009), Mobilitetsrapport 2009, Bergen and SIU (2009), Tilstandsrapport 2008 Internasjonalisering av norsk høgre utdanning, Bergen. 134 NIFU STEP (the Norwegian Institute for Studies in Innovation, Research and Education) has worked with evaluations of the Leonardo da Vinci, Socrates and Erasmus programmes in Norway in 2001 and 2007, the main focus in both reports is on student participation, see: NIFU (2001), Norwegian participation in the ERASMUS programme, Oslo and NIFU STEP (2007), Evaluation of the Leonardo da Vinci and SOCRATES programmes in Norway, Oslo. Central findings in the 2007 report are: the programmes had a significant impact on internationalisation of education and vocational training in Norway, simplification and flexibility are important especially for encouraging participation from marginalised groups and small- and medium-sized enterprises, better collaboration between the beneficiaries and the NA (as well as the NA and the EC) is necessary, and the involvement of the members of the national programme committee should be strengthened. 135 See: Norwegian Academy of Music (2008), Internasjonalt Toneangivende Norges musikkhøgskoles strategiske plan , Oslo. SIU mentioned the Academy of Music as best practice example in its 2008 National Report on Erasmus. 171

172 2.11. What is the European added value? Would other national schemes / instruments provide enough support to the activities funded by the evaluated LLP sub-programme / action, if there was no LLP? Would adequate attention be given to the challenges faced by the disadvantaged citizens in meeting the lifelong learning needs? Erasmus Teaching Abroad is a very important tool for Norwegian higher education institutions and Norwegian society and has a high European added value. It supports the European dimension of education in Norway and the international policy of higher education institutions and the Norwegian government. Even though Norway is not a Member State of the EU the Ministry of Education and Research participates in the LLP committee and the Bologna process, which is important for the Ministry, as all countries face similar challenges and Norway is described as a small country which needs exchange with others to further develop its policies and organisations. Norway and the other Nordic countries used the experience gained with the Socrates programme (Erasmus was a part of Socrates until 2006) for developing Nordplus 136, a programme which also contains funding for mobility grants for teachers. Nordplus is seen as complementary to Erasmus and could not be a substitute as it covers only the Nordic and Baltic countries. There are no national schemes in Norway which provide support for teaching abroad. Erasmus Teaching Abroad is an important tool for teacher mobility (outgoing and incoming). It is possible that some higher education institutions might invest in teacher mobility without Erasmus but this is highly hypothetical and the interviewees themselves could not provide any concrete information. It is impossible to estimate what the amount of funds available or the percentage rate for teacher mobility would be without Erasmus What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? The earlier mentioned Quality Reform in Norway led to the establishment of one national agency for Internationalisation (SIU, since 2007). This in addition to the creation of one umbrella programme (the LLP) for the several European education programmes created synergies between the different sectoral programmes of the LLP (Comenius, Erasmus, Leonardo da Vinci and Grundtvig) and overlaps are easier to avoid. Synergies inside the sectoral programmes are visible: teacher mobility is, for example, linked to student mobility. Teachers who have used Erasmus can better convince their students to go abroad than teachers who have not been involved in international exchange. Therefore teacher mobility is very important as the number of students, who use Erasmus mobility grants (outgoing), is decreasing in Norway. Also synergies with staff mobility can be assumed: SIU has worked with an evaluation of staff mobility in recent months. higher education institutions which used staff mobility are also active users of teacher mobility. In addition, the evaluation shows that persons, who used staff mobility grants, were very satisfied. 138 Nonetheless, Erasmus Teaching Abroad (outgoing) is used very little for creating links between higher education institutions and enterprises, which is a matter of regret for members of the National Programme Committee Erasmus representing the social partner organisations. 139 Cooperation with enterprises could be promoted and encouraged; information material for enterprises could be developed. 136 Nordplus Adult, see: On 19 th May, SIU confirmed on the phone and in writing that it would be highly hypothetical to guess if and how much money institutional and/or individual beneficiaries would invest in teacher mobility, if Erasmus Teaching Abroad grants would not be available. Also institutional beneficiaries could not answer this question. 138 SIU (2010), Ansattmobilitet i Erasmusprogrammet en kartlegging av aktivitet og effekter for mobilitetsopphold gjennomført i 2008, Bergen (not published yet, May 2010). 139 The organisations represented in the National Programme Committee Erasmus are: the Norwegian Association of Higher Education Institutions (UHR), represented by University Colleges in Bergen (deputy Ålesund), Oslo (deputy University of Bergen) and Volda (no deputy), and social partners, represented by Unio (the Norwegian Confederation of Unions for Professionals), Akademikerne (the Federation of Norwegian Professional Associations), LO (the Norwegian Confederation of Trade Unions), NHO (the Confederation of Norwegian Enterprise), STL (the Norwegian Association of Students) and NSU (the National Union of Students in Norway). The Norwegian Ministry of Education and Research is an observer. The function of the Committee is to secure Norwegian participation in the programme, be involved in the approval of applications, support the dissemination of results and advise the NA. See: 172

173 Higher education institutions need an international strategy, in which exchange of students, staff and teachers and cooperation with enterprises is included to create synergies between different actions of Erasmus EFFECTIVENESS 3.3. Effectiveness in achieving targets and contributing to priorities: What is the progress in achieving action / sub-programme targets against financial progress of the programme? What is the extent to which the beneficiaries are satisfied with the outputs and results of different sub-programmes and actions? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The Erasmus Teaching Abroad (outgoing) action is implemented in an effective way in Norway, in particular at University Colleges, where the number of mobility grants for teachers (outgoing) is increasing. Organisations which use Erasmus Teaching Abroad grants are very often host organisations for teachers from other countries using Erasmus. The organisations have different procedures concerning the incoming teachers: some have an overview of how many teachers are visiting their organisation, others do not. Experience with incoming teachers is not documented and evaluated. In one case, the fact that the incoming teachers create no costs for the host organisation was used as argument for working with Erasmus. As stated above, the NA receives no information from institutional beneficiaries about incoming teacher mobility. The NA aims to use the financial resources available for Erasmus Teaching Abroad and has in recent years used more than the indicative rate of 10% for teacher and staff mobility. Still SIU had to transfer some money back to the EC because of over-estimates made by some of the institutional beneficiaries. Competition is not a topic for SIU or institutional and individual beneficiaries, as collaboration is based on mutual understanding and all teachers, who provide a complete and correct application, can receive a mobility grant. SIU hopes to have better control in the future, e.g. through close cooperation with the institutional beneficiaries. SIU is also considering if the flat rate ( 800, 900 for individual beneficiaries in Northern Norway because of higher travel costs), which is given for each mobility grant, is too low, but has no evidence for this. The flat rate can be used for travel and subsistence costs (including costs for overnight stays); mobility can take between one day (including at least five hours teaching) and six weeks. Very often the flat rate does not cover all costs and is co-funded by the institutions. If the mobility is cheaper than 800 the money not used has to be paid back to SIU. The interviewees at higher education institutions estimated that at least 75% of all individual beneficiaries are satisfied. Each individual beneficiary fills out a form reporting about the mobility 141, which she/he delivers to the contact person in the administration at her/his organisation, who handles the mobility grants. This report stays with the organisation and is not delivered to the NA; it is the choice of the organisation to use or evaluate it or not. During the in-depth interviews the institutional beneficiaries mentioned the following obstacles with Erasmus Teaching Abroad (outgoing): - the heavy workload of teachers and the lack of substitution during absence; - lack of time because of part-time positions; - possible applicants are more interested in research than in teaching; - the lack of long-term planning; - bureaucracy (e.g. the individual beneficiaries have to fill out four different forms, some of which contain the same information); - the lack of recognition for participation in teacher exchange; - the lack of the required and adequate language competence (see ) SIU came to similar conclusions in its report about staff mobility: SIU (2010), page The form contains the following information: name; host organisation; period; information about the contact person at the host organisation; teaching hours, methods, language, subjects and material; experience and impact. 173

174 Higher education institutions are working to overcome these obstacles in different ways by: - promoting information and awareness through an Erasmus Partner Week 142 or International weeks ; - providing extra hours for teachers preparing their Teaching Abroad mobility; - establishing joint programmes with organisations in other countries; - involving students and teachers in mutual exchange and organising group visits; - collaborating with other higher education institutions in the own country to support the development of international exchange and to facilitate visits from foreign partners; - establishing international coordinators at faculty level, who can use 20% of their working-time for internationalisation, which creates positive environments for international collaboration and supports Erasmus Teaching Abroad actions (incoming and outgoing); - organising for example, as head of the international office regular meetings with the international coordinators from the different faculties; - visiting every year one of the partner organisations in a European country to improve collaboration between the organisations (this visit should be made by the person who is responsible for Erasmus at the administrative level at the higher education institution) What are the main external factors affecting the outputs and results of the sub-programme / action? There are clearly external factors influencing the use and impact of Teacher Mobility: some obstacles are created by different education systems, curricula and time frames. For example, countries involved in Erasmus have different time frames for academic years and teaching periods at universities and university colleges. For using Teacher Mobility in an effective way and to increase possible impacts, it would be useful to create exchange projects in which students and teachers - and maybe administrative staff working with internationalisation - from different countries are involved. The project participants could find solutions for handling external factors like the different time frames. Erasmus Teaching Abroad grants (outgoing and incoming) could be used as starting grants for such projects. The capacity and willingness of institutional beneficiaries in Norway differs a lot; in general it can be stated that university colleges are more interested than universities (see 2.2.) and that cooperation with enterprises is not visible or is missing. Especially the interviewed committee members representing the social partner organisations underlined the necessity for leadership to be behind the activity and to promote international collaboration. This could be achieved through involving organisational and department/faculty leaders in international mobility. All interviewees mentioned that the marketing of grants should be improved, for example, through virtual social networks and at enterprises. All actors should be involved in marketing and dissemination the national ministry, the NA, the committee and the institutional and individual beneficiaries. An obstacle in Norway seems to be the low level of interest in exchange with organisations inside the EU; historically there are close collaborations with the United States (US) and people (students and staff) are interested in exchange projects with the US or Australia. 143 Many people are sceptical about the EU and very often knowledge about the EU Member States, especially the countries which joined the EU in 2007 (Romania and Bulgaria), is lacking. This explains partly why the host organisations are mainly in EU15 Member States like Germany ( : 61 grants), the United Kingdom ( : 25 grants) and Spain ( : 21), even though the sending organisations have cooperation agreements with many organisations in many different countries. 144 In no Teaching Abroad grant was used for visiting organisations in Cyprus, Romania or Slovenia. The financial crisis will probably become another external factor affecting Erasmus grants it can be assumed that teachers and organisations facing economic problems will be less interested in international exchange (as host and as sending organisations). This is not documented yet. Environmental factors will influence the use of mobility grants: in recent weeks (April and May 2010) the use of Erasmus Teaching Abroad grants has been affected by the ash-clouds, caused by the active volcano in Iceland. Individual beneficiaries could not travel and cancelled their mobility or had to stay longer in one of 142 See also presentations at an Erasmus Partner Week: See for example: SIU (2010), Hvorfor sudere I utlandet? Norsk studenters motivasjoner og barrierer for å ta et studieopphold i et annen land en kvantitativ analyse, Bergen, page All interviewed organisations have more than 40 bilateral Erasmus agreements. 174

175 the host countries. This will influence the number of mobility grants in and maybe also during the next years, as it is not predictable how the volcanic eruptions will develop in the future. Persons in Norway are more dependent on air travel than persons on the European continent because of the size and location of Norway; in addition the Norwegian railway system is underdeveloped compared to countries like Germany or Spain Effectiveness in implementing horizontal policies: How does the sub-programme / action contribute to the horizontal priority of multilingualism? Based on the available data and the collected information (through desk research and interviews) it can be assumed that Erasmus Teaching Abroad encourages multilingualism 145, even though language problems were described as one of the biggest obstacles and as one of the reasons why teachers do not participate or why they are mostly collaborating with organisations in, for example, Germany, Spain and the United Kingdom (see ) German, Spanish and English are common second languages in Norway. Most likely teachers with no or very little knowledge of foreign languages do not participate in the Erasmus Teaching Abroad actions. 146 Statistics provided by SIU show that languages and philological sciences are fifth on the list of subject areas of outgoing teacher mobility (from 2000 to : 193 grants); in first place is education and teacher training (450), second is art and design (437), third medical sciences (329) and fourth social sciences (261). Innovative or good practice examples are not disseminated and dissemination activities are not satisfactory, all institutional beneficiaries and stakeholders could improve their dissemination strategies Is the implementation of equal opportunities (EO) satisfactory in the decentralised actions of the programme? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? Equal opportunities issues could be improved, even though they are guaranteed by Norwegian law. 147 The application forms and information material used by the individual beneficiaries could refer to these laws and the horizontal policies in the LLP making people more aware of equal opportunities. SIU recommends that higher education institutions promote mobility for teachers with disabilities, but has not received applications from this target group. 148 Specific incentives for persons with disabilities are not available. More men than women participated in in outgoing teacher mobility: 42,2% were women and 57,8% were men 149. At the same time the university colleagues, which are the main users of teaching staff mobility, have more female than male teachers (in 2008: 51% women, 49% men). 150 Maybe this mismatch could encourage all stakeholders to discuss equal opportunities. The interviewee at the Ministry of Education and Research suggested using equal opportunities as a guiding principle for the further development of international strategies. The above mentioned SIU evaluation on Erasmus staff mobility contains information about the average age of teachers using Erasmus Teaching Abroad grants: in 2008 the average age was 51 years (50 for women and 51 for men) 151 ; including more younger teachers could therefore be one objective for the future. As stated above, detailed information about incoming teachers (for example about their gender, age and profession) is not available. 145 See: Det Kongelige Kunnskapsdepartmentet (2008), St.medling.nr. 23 ( ), Språk bigger broer Språkstimulering og språkopplæring for barn, unge og voksne,oslo. Erasmus is mentioned as one of the programmes which supports multilingualism, see page In-depth evaluations are missing. It could be interesting to evaluate the reports written by the beneficiaries to find out more about their language competence (before and after the mobility). 147 The Act relating to Gender Equality shall promote gender equality and aims in particular at improving the position of women. Women and men shall be given equal opportunities in education, employment and cultural and professional advancement (amended by the Act of 14 June 2002 No. 21, in force from 1 July 2002). The Anti-Discrimination Act came into force 01 January 2006: the purpose of this Act is to promote equality, ensure equal opportunities and rights and prevent discrimination based on ethnicity, national origin, descent, skin colour, language, religion or belief. The Act shall apply in all areas of society except for family life and personal relationships. 148 Reported by SIU during the interviews and in the 2008 National Agency Report. 149 Figures provided by SIU, 11 May See: and Statistics Norway (2009), Facts about education in Norway 2010 key figures 2008, Oslo, page SIU (2010), page

176 3.3. Effectiveness in integrating previous activities into the LLP: Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? In general the assembling of the education programmes under one umbrella programme (the LLP) is seen as very positive and successful, specifically as one NA (SIU) is handling the programme. 152 It was also reported that the application process has become easier, which made the measures more attractive. The use of Erasmus Teaching Abroad grants in recent years shows the growing attractiveness. The figures for Teaching Abroad (outgoing) from the recent years are as follows: : 308, : 294, : 346, : 345. The figures for incoming teachers are as follow: : 300; : 321; : 367; : not available yet. SIU and the interviewee at the Norwegian Ministry of Education and Research described the participation as going in waves and does not see major problems with the decrease in Teaching Abroad (outgoing) in One obstacle mentioned by SIU and members of the Erasmus national committee is that potential individual beneficiaries are still confused when they hear about the different parts of the LLP and sometimes people confuse Erasmus and Erasmus Mundus. The NA mentioned that it would be better to use the name Erasmus only for one (sectoral) programme and not in combination with other programmes like Erasmus Mundus and 'Erasmus for young entrepreneurs' Effectiveness in reaching the target groups and disseminating results: How successful is the LLP in reaching the target groups and institutional beneficiaries of subprogrammes? SIU holds a yearly conference presenting the programmes, frameworks and achievements and sees the conference as a very important tool for developing cooperation with the institutional beneficiaries. 153 The participation in the programme can be described as fair : Norway has 65 registered higher education institutions (public and private, figures from 2006) and in slightly more than 50% (34) used Erasmus Teaching Abroad (outgoing) grants. 55 of the 65 higher education institutions hold an Erasmus University Charter, which is the pre-condition for participating in Erasmus. It is positive that all Norwegian regions are represented among the active higher education institutions. Information and dissemination could be improved at national and institutional levels; in the case of university colleges and universities at central administration and faculty levels. Leaders at higher education institutions should be encouraged to support Erasmus Teaching Abroad grants and to create environments where people want to participate in exchange; for example, by discussing and facilitating substitution and establishing rewards for international engagement To what extent are the Programme results adequately disseminated and exploited? The implementation can be described as adequate and - as already mentioned - dissemination activities and marketing could be improved at all levels. higher education institutions are working with the results achieved through teacher mobility (outgoing and incoming) in different ways; some use it for presentations at international weeks, external and internal seminars or as good practice examples for their internationalisation. Some interviewees described a lack of suitable information and the interviewed committee members representing the social partner organisations wished for more evaluation and dissemination. SIU participates at institutional conferences and seminars and reports about Erasmus. SIU has worked with an evaluation of staff mobility (see 2.4.) and has been encouraged by this case study to evaluate teacher mobility. 152 This was confirmed by the interviewees at the Ministry of Education and Research and the beneficiaries. 153 See SIU s website: 176

177 4. EFFICIENCY 4.7. Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? The management can be described as well-functioning, interviewees praised the staff members at SIU, and SIU had no complains about institutional and individual beneficiaries. SIU is monitoring the implementation and would be interested to see the individual reports written by the individual beneficiaries (the reports are kept by the organisations). SIU uses monitoring visits to provide information about Erasmus and to discuss the different grants available. Improvements could be achieved in publishing the selection criteria for applications: no award criteria are set at European level 154 and the national selection criteria are not published. Criteria named by SIU during the interviews are past performance (reports from previous years and interim reports) and quality of the application (including concrete explanations concerning the number of mobility grants) Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? The management is well functioning and nobody complained about late payments. SIU and the interviewees at beneficiary institutions and the Ministry of Education and Research described the administrative tasks as too time-consuming: too many forms have to be filled out. Sometimes the different forms contain the same information; this could be collected in one form. For example, in addition to the documentation of travel and accommodation costs the outgoing teachers have to deliver three different forms after their teaching abroad visit: the Teaching Assignment (filled out by the teacher, the host and home institution), the Confirmation of Erasmus Teaching Assignment (filled out by the host institution) and the Report (written by the teacher); it was proposed to assemble the Teaching Assignment form and the Confirmation of Erasmus Teaching Assignment in one form. SIU would like to have more flexibility, for example to transfer Erasmus grants which are not used by student mobility to teacher mobility. Even though SIU transferred back to the EC grants not used for teacher mobility, they would like to have more grants available for teacher mobility, which would encourage higher education institutions to improve these activities. SIU proposed that the EC could organise training seminars for the new management tool LLP-Link ; the tool is difficult to handle and could be explained during the training seminars. In addition a manual could be useful, in which it is described how to work with LLP-Link and which explains why and how to register specific information. Neither SIU nor the interviewed institutional beneficiaries use external assistance for working with Erasmus; they only have external auditors (as requested) Is the monitoring information used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? SIU uses the interim reports sent by the institutional beneficiaries as a monitoring tool: the interim reports show how many grants are used and make it easier to predict the future needs for grants. Therefore the information provided in these reports allows adjustments during each mobility/academic year. SIU uses the interim and final reports from higher education institutions for statistical purposes and the reporting to the EC; the reports are not used currently for other documents or evaluations. As stated above, information, evaluation and dissemination should be improved at all levels to encourage an increase in teacher mobility (outgoing), which would have a positive influence on the other activities, like incoming teacher and outgoing and incoming student and staff mobility. 154 See: LLP 2010, Fiche N /File Nr 19: ERA-Mob p

178 11. Comenius Bilateral and Multilateral Partnerships : Italy 1. INTRODUCTION The overall purpose of this case study is to provide an in-depth evidence-based insight into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme s action. The information presented in this case study is based on the following sources: - In-depth interviews with the national agency, the national authorities and the institutional beneficiaries of the Comenius Bilateral and Multilateral Partnerships action. In total 8 persons have been interviewed to support the case study (the full list of interviews is presented in Annex 4); - Desk research, which sought to analyse national legislation, the implementation statistics of the action and other documentation, which was provided by the national agency, the national authorities and the institutional beneficiaries. The interviews and desk research were carried out during the period of 26 April 31 May RELEVANCE The main objective of this section is to describe the European added value of the selected action To what extent the intermediate and specific objectives of the action remain pertinent to the national policy priorities? Comenius School partnerships aim at enhancing the European dimension of education by promoting joint cooperation activities between schools in Europe. The projects give pupils and teachers in different countries an opportunity to work together on one or more topics of mutual interest. School partnerships help pupils and teachers to acquire and improve skills not only in the topic or subject area on which the project is focused, but also in terms of teamwork, social relations, planning and undertaking project activities and using information and communication technologies (ICT). Participating in a partnership with schools from different countries also gives pupils and teachers the opportunity to practise foreign languages and increases their motivation towards language learning. The specific objectives for the Comenius School Partnerships (both Bilateral and Multilateral) remain relevant to current Italian national priorities in the context of lifelong learning. Neither the LLP NA nor the National Authority mentioned any important objectives which were not addressed. The specific objectives for Comenius Partnerships have a broad meaning and the institutional beneficiaries can always customise a partnership according to its needs, under the fixed objectives of the Comenius Partnership. In particular, both BSP and MSP can act as experimentation laboratories, in relation to developing innovative practice within the general educational sector. Although the school partnerships would not necessarily do this, in Italy they often provide experience for both students and teachers of new ways of learning and teaching. The institutional beneficiaries interviewed have reported evidences to support this point: the direct/indirect comparison among teachers from different schools and different countries stimulates analysis and re-thinking of the own teaching and learning approach. Moreover, in the specific case of Italy, the LLP NA (for Comenius) operates within the context of INDIRE ANSAS, an institutional agency for the innovation and development of school autonomy 155, that is formally an actor in the national educational system and under the control of the MIUR, the National Education Authority. 155 The Italian school system has been characterised, in the last decade, by a process of granting schools a degree of autonomy in terms of educational, managerial and financial functions. Autonomy goes hand by hand with responsibility and accountability systems of schools. The School Autonomy Reform (SAR) is based on the assumption that more local managed schools will improve the overall performance, through more autonomy, responsibility and accountability. 178

179 This specific solution has been the precondition to the achievement of positive results in the field of innovative learning experiences, in case of Comenius Partnership participation too. Avoiding confusion on this issue, it is necessary to underline that the school autonomy in management do not imply the development of autonomous learning ability in students. But an innovative vision could be transferred by the formal organisation to the different institutional beneficiaries and then to the students To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? The objectives of the Comenius sub-programme 156 for the specific target of the BSP & MSP, as reported by the interviewed beneficiaries, remain pertinent to the needs of teachers and schools. This would appear to be confirmed by the high level of applications for partnerships at both primary and secondary school levels. For the pre-primary sector, the lower number of applications could reveal a difficulty in transferring the COMENIUS Partnership objectives to their younger target groups. Moreover the specific needs related to the learning and teaching of foreign languages in this sector are lower than the upper levels: there are lower levels of foreign language expertise amongst teachers and the education offer to the pupils is less centred to foreign languages. Although the statistics shows a decreasing trend in the number of applications, this should not be misunderstood. A lot of institutional beneficiaries every year are the so called newcomers schools and education institutes that have not previously submitted any applications under LLP Comenius Partnership. Moreover new partnerships under LLP from 2007 have been for two years and frequently an institutional beneficiary is not able to manage more than a single Comenius Partnership every two years. Then it is only from 2009, but especially in 2010, that the institutional beneficiaries could have re-applied for a new Comenius partnership. The priority of the LLP NA is given, between two beneficiaries levelled in term of quality of the partnership, to new-comers. As a regular procedure, this selection criterion has been adopted to maintain a constant turnover among the beneficiary schools. Another element to be considered is that during the transition year 2007 some previously approved partnership under Socrates Programme have been renewed for 1-2 years and are included in the total amount of MSP. Moreover, the peak in the number of applications for the year 2007 could have been related to the higher promotion and communication effort spent at the launch the new LLP. When the emphasis decreased and the attention phase became normality a reduction of applications has been registered. So the apparent decreasing trend could not represent a reduction of the alignment between Comenius objectives and target groups needs, and a real comparison, concluding, will be possible including the data of the entire Finally, other aspects support the continued relevance of Comenius partnerships to schools: - Most beneficiaries have not been able to involve all their students and teachers in the partnership and in mobility. So the objectives continue to remain pertinent for these target groups. - The majority of the beneficiaries in 2009 and 2010 have applied for a new Comenius Partnership. - The Comenius Partnership application has often been based (at least partly) on a foreign language learning objective. This is always relevant to students learning needs and the experience of the Comenius Partnership is directed to new target students every time. - The entire specific objective could be reached in a more advanced way (for example multicultural experiences, ICT enhancement, sharing of educational methodologies among different countries than previous one). 156 Specific objectives of the Comenius programme shall be: (a) to develop knowledge and understanding among young people and educational staff of the diversity of European cultures and languages and its value; (b) to help young people acquire the basic life-skills and competences necessary for their personal development, for future employment and for active European citizenship. The operational objectives of the Comenius programme shall be: (a) to improve the quality and to increase the volume of mobility involving pupils and educational staff in different Member States; (b) to improve the quality and to increase the volume of partnerships between schools in different Member States, so as to involve at least 3 million pupils in joint educational activities during the period of the programme; (c) to encourage the learning of modern foreign languages; (d) to support the development of innovative ICT-based content, services, pedagogies and practice for lifelong learning; (e) to enhance the quality and European dimension of teacher training; (f) to support improvements in pedagogical approaches and school management. (Official Journal of the European Union, DECISION No 1720/2006/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL). 179

180 2.3. The European added value. Would other national schemes / instruments provide enough support to the activities funded by the Comenius Partnerships, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? The European added value in the Comenius School Partnership is very high in the Italian context. The NA and the National Authority (MIUR) recognise this aspect as one of the most important results of the Comenius sub-programme and, in particular, of the School Partnership actions. Sometimes the National Authorities used the term overbooking referring to the participation of Italian schools in the Comenius subprogramme. The numbers of the applications and of funded projects make this very clear. The beneficiaries recognise the great added value of the European opportunities, in particular of the Comenius Partnership they took part in, confirming in their partnership final report and during the interviewees, that probably none of the performed activities of their projects would be implemented without LLP support. Sometimes the partnership participation and the European vision of a school became a strength presented during the initial orientation of the potential students. Nevertheless, some programmes funded by the Italian national education authority support the continuous professional development of school teachers. An example is the financial support for foreign visits aimed at enhancing the language skills of teachers of foreign languages other than English (French, German, Spanish, etc). Another programme is related to the implementation of ICT in schools. These programmes aimed to achieve results which may also indirectly reach the beneficiaries through their Comenius Partnership Projects. Moreover the European dimension of Comenius Partnerships is recognised by all those interviewed and supported by the documents acquired in the desk research, especially in term of enhancing cooperative attitudes and skills among teachers and students involved. Another aspect is that in Italy often the schools suffer from the reduced availability of national resources for in-service training/professional development and the European LLP is a real opportunity for lots of these educational institutions. Finally, if, hypothetically, the LLP would not have been established, probably the following activities would have been affected by a reduction in effectiveness, in the case of lack of alternative support by national policies: - Meetings, discussions, etc. between institutions of Participating Countries; - Development of common EU-wide policy tools (indicators, databases of good practice, technology), though this is not only related to Comenius Partnerships; - Introduction of European dimension in the national systems of education and training; - Funding for mobility of individuals provided by national, regional or local authorities What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? In the context of Italian LLP programme there are two different LLP NA, one for the Leonardo projects and one for the rest of the LLP sub-programmes (including Comenius). This particular situation meant that, from the beginning, there had to be a joint collaboration and a synergies development process: for example the LLP programme has a unique website 157 for all the different sub-programmes. All the orientation events held by the LLP NA are related to all the sub-programmes and potential beneficiaries are invited to follow specific sub-programmes/action according to their needs. Moreover common administrative services (especially for non-leonardo sub-programmes) are managed by the same institution and the same administration. On one hand, some previous barriers among different sub-programmes have been broken down by the operation of the LLP as a unified programme, allowing similar activities under different sub-programmes and enhancing flexibility. On the other hand, there is the risk of duplicating the names for the same action, revealing difficulties in finding specificities (for example Grundtvig Workshops and Comenius Seminars)

181 From the beneficiaries point of view the existence of a unique Lifelong Learning programme seems to be only formal, and only in a few cases were the visibility of the other sub-programmes, or any synergies between them, mentioned during the interviews. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The overall progress in achieving targets and the general effectiveness in reaching the objectives is high. During the period , in Italy, more than applications have been submitted to the National Agency by the Italian potential beneficiaries. Approximately half of the proposals have been approved and funded. In 2007 the official data reveal a peak in the number of both applications and approved Comenius partnerships. Detailed quantitative data of the Bilateral and Multilateral school Partnership (BSP&MSP 158 ) in Italy are reported in the table below. Table Applications, approved Partnership and total funding in Italy Comenius Bilateral and Multilateral School Partnership in Italy Year Applications Approved Partnerships (BSP+MSP) Total funding TOTAL Source: LLP National Agency, 2009 Figure Applications and approved Partnership in Italy (LLP National Agency, 2009) Applications Approved partnerships (BSP+MSP) Source: LLP National Agency (2009), LLP come va? Anno 2009: riflessioni di metà percorso Comenius, Erasmus, Grundtvig e Visite di Studio - Comenius: impronte di cambiamento nella scuola., Florence Just under 50% of this huge number of applications have resulted in funded partnerships, taking 2007, 2008 and 2009 together. As reported in the Table below, a considerably lower success rate is recognizable for Bilateral Partnership applications, especially in If the success rate of a Multilateral School Partnership application is around the 50%, in 2007 the Bilateral Applications success rate is 18%, increased to 37-39% during the period Generally, as in the case in all EU Member States, the absolute number of Multilateral School Partnership is much higher than the bilateral one. In Italy, during the 2009 the approved BSP number is limited to 67, against 532 MSP. 158 BSP & MSP is the acronym for Bilateral School Partnership and Multilateral School Partnership. 181

182 Figure age distribution of BSP and MSP (LLP National Agency, 2009). 100% 80% 60% 40% 20% 0% BSP MSP Source: LLP National Agency (2009), LLP come va? Anno 2009: riflessioni di metà percorso Comenius, Erasmus, Grundtvig e Visite di Studio - Comenius: impronte di cambiamento nella scuola., Florence. Table Applications, approved Partnership and % rate of BSP and MSP in Italy BSP MSP TOTAL Applications Approved % Applications Approved % Applications Approved ,18% ,12% ,92% ,45% ,78% ,27% Source: Elaboration Istituto Italiano di Valutazione on data from LLP National Agency These data reflect a constant trend in the participation of Italian beneficiaries to the application process for a Comenius Partnership, although the first year of LLP implementation a larger amount of MSP have been submitted and approved. Moreover the majority of the beneficiaries prepared (or are preparing, a new Comenius Partnership application for the This reflects also a high level of general satisfaction by the beneficiaries (schools, principals, teachers, students and pupils families) for the closed projects. All the interviewed beneficiaries confirmed that and both the LLP National Agency and National Authority are aware of that. In term of influence that the participation in Comenius Partnership had on the beneficiary s organisation, the most frequent aspects are: - Sharing of educational good practice on a European level: especially in case of partnerships focused on specific target groups with specific needs the sharing of methodological approaches and good practices could be higher. Nevertheless, all the beneficiaries interviewed mentioned positive results in this area and the LLP NA too. - Strengthening of educational partnerships and networks: most of the beneficiaries remain in contact with the partner schools all around the Europe and in some cases new Comenius Partnership projects have been presented together; - Strengthening of the quality of in-service training and professional development for serving teachers; - Improved language skills among teachers and pupils: in the specific case of English language, all the beneficiaries revealed a high satisfaction in enhancing their spoken English to communicate with school partners. Not only teachers of English as a foreign language cited this result but also teachers of other disciplines. Moreover a basic knowledge of other different languages is often recognised, especially for teachers/students involved in Bilateral School Partnership (see par for details). - Improved European links and interaction between Local Authorities, schools, school education organisations, business and industry: the scenario revealed in Italy is a distributed map of beneficiaries with specific educational, social and economic needs, including often schools and educational institutions in very small towns. In this socio-economic context the experience of a Comenius partnership (of 24 months, for example) represents often a great event for all the social community and the whole town. In many cases the mayor, the local administration, and private firms have been involved in the preparation/implementation of the Comenius Partnership. - Improved ICT skills in both teachers and students: this is simple but true, the experience of a Comenius Partnership often represents a way to use ICT tools in the context of education system. Nobody used to write s before the Comenius, now all of us have Skype on our computer (An interviewed institutional beneficiary). 182

183 The most important issue to explain the satisfaction of the beneficiaries (pupils and teachers) is the recurrent request for new partnership applications. Another issue is the maintenance of contacts between schools participating in the same Comenius partnership after the end of the funded project. Concerning the satisfaction of the beneficiaries, all the interviewed persons declared their high satisfaction with the achievement of obtained outputs and results. In some cases the assessment of the satisfaction has been based on data collected with questionnaires submitted to teachers/students/pupils families. This is the same point of view expressed by the LLP NA. In achieving the interim objectives, the total number of persons in mobility during the period reached the 39,000 persons (among teachers and pupils). Every year 10,000-14,000 persons travelled all around the Europe under the Comenius partnership projects and reported their complete satisfaction. The following table reports the annual data. Table Persons in mobility under COMENIUS Partnership in Italy (LLP National Agency, 2009) Year Preparatory visits MSP mobility BSP mobility Total TOTAL Source: LLP National Agency (2009), LLP come va? Anno 2009: riflessioni di metà percorso Comenius, Erasmus, Grundtvig e Visite di Studio - Comenius: impronte di cambiamento nella scuola., Florence What are the main external factors affecting the outputs and results of the action? The most important external factor for the Italian context is school autonomy. In the Italian educational system there are sometimes rigid rules but every school and every principal - has significant autonomy, in order to enable them to better manage their resources (time, funds, teachers, etc). On one hand, some rigid rules, related for example, to the difficulties in giving credits to the teachers effort spent for the Comenius Partnership coordination/participation, affect negatively the output and results in some cases. On the other hand, school autonomy guarantees the alignment of Partnership objectives in the whole school educational programme (POF). School Autonomy allows also the correct management of a specific national line of funding 159 dedicated to each school and based onto the number of persons (staff, teachers, students) enrolled in the school each school year. As reported by all beneficiaries interviewed, economic factors sometimes affect the partnership s results. This is also reported by both NA and National Authority for education (MIUR). More specifically, the funding of Comenius Partnerships are determined only by the number of exchanges and a minor part of the funding can be used for other purposes such as dissemination costs, materials, etc, but the entire effort allocated to the partnership is not formally covered. The single beneficiary is responsible for the management of the complementary effort spent in preparation/coordination of the mobility, organisation of dissemination events, and provision of needed materials and so on. In some cases contributions from the local municipality or from private firms have been strongly asked and obtained, in order to cover at least part of the other expenses related to the Comenius partnership management. A key success factor for the achievement of the partnership good results is the internal alignment of visions between the school manager ( Dirigente Scolastico in Italy) and the teacher/s responsible for the Comenius Partnership. This element should be taken into high consideration and, in case of non alignment; it could become a negative factor affecting the partnership. It is difficult to find validated best practice on this, but the experience says that the more internal alignment a beneficiary have, the more successful can be the partnership. 159 Fondo dell Istituto is managed by the single Italian school with an internal agreement among staff and teachers. 183

184 3.2. Effectiveness in implementing horizontal policies How does the action contribute to the horizontal priority of multilingualism? The horizontal priority of multilingualism is partially in contrast with the national approach to single foreign language knowledge (mainly English). Nevertheless, BSP & MSP allowed the experience of practical learning of other different and often previously unknown languages: the overall contribution of Comenius Partnerships on this priority is medium to high. Not only English but also other European languages have been learnt by the students and the teachers involved in Comenius BSP & MSP. Although the main spoken language in the Comenius Partnership is commonly the English language, students and teachers taking part in mobility activities) or partnership meetings often learn a minimum base of grammar and spoken rules of the country language of their partners. In this sense the BSP is more advanced than the MSP, in supporting a deeper learning of another European language. In many cases of BSP a lot of effort has been spent to support the multilingualism priority, although some difficulties emerged involving competent foreign language teachers in case of not so popular languages. Basing the finding on the answers provided by LLP NA and institutional beneficiaries, the interviewees gave assessed as more successful approach in contributing to the horizontal policy of multilingualism, the integrated approach, when policy is implemented through priorities set in separate sub-programmes or annual Call for Proposals. All these issues could be collected in a clear recommendation for the further LLP period, proposed by the Italian LLP NA. Resulting that Bilateral School Partnerships contribute to multilingualism much more than MSP, the total funding for Bilateral Partnerships at national level could be increased for the period and further post-2014 period How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? The overall finding regarding implementation of equal opportunities appears satisfactory. Although there have not been specific priorities on BSP and MSP, the implementation of horizontal policy related to equal opportunities is embedded in all LLP sub-programmes and especially in Comenius Partnerships. Specific attention is required in the application for a new Comenius Partnership and all the Applicants are required to include a specific detailed objective related to gender issues and equal opportunity policies for the involvement of males/females in the partnerships. This issue is specifically covered by the definition of equality criteria for the selection of students/pupils for mobility activities among all the interested people in the schools. All the interviewed beneficiaries reported the particular attention given to the selection of an equal number of males and females for the mobility, as well as of different ages (if applied). Sometimes priority is given to particular cases of disadvantaged students (first time travelling abroad, or students in difficult social conditions). Concerning the equal gender representation among the teachers selected for the mobility, the lower number of male teachers is an important factor (in Italy the number of females teachers is much higher than the number of males). On the side of ethnicity or disability the Italian experience reveals a lot of partnerships focused on these specific target groups, in parallel with above mentioned specific selection criteria for the participants in the exchanges Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? The overall finding regarding the effectiveness of the LLP integration in Italy is not assessed at high level. In the Italian context the education sector and the training sector are institutionally part of two different Ministries: education under the MIUR Ministry of education, University and research - and training is under the competences of MLSPS Ministry of welfare, health and social policies. The National Authority for education (MIUR) revealed a position not in favour of the integration of activities within the LLP, and expressed a wish for education and training activities to be managed separately. This 184

185 would more accurately reflect the actual situation of the Italian institutions. It seems to be not a Europe-wide consideration, but only justified by the specific Italian situation. On the side of institutional beneficiaries the visibility of the whole LLP is limited, because of their specific involvement in Comenius partnerships. In some cases the beneficiaries had an experience of a Comenius partnership under the previous Programme and had not experienced any significant differences after the implementation of LLP in On the side of LLP NA one of the most important services they are managing is the orientation of potential beneficiaries among the LLP different sub-programmes and opportunities, based on the beneficiary s needs. Specific orientation events have been held all over the national territory by the LLP NA before the call for proposals in order to present and orientate the potential applicants. It is opinion of the evaluator that they were able to work more efficiently in this, given the integrated approach. Moreover, from the beneficiaries point of view, a single programme could be more easily recognised by the target groups in different phases/ages of their lifelong learning process Effectiveness in reaching the target groups and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? The various clusters of target groups and related approved partnerships are reported in number in the table below, underling the success in reaching different targets in the period Table Approved partnerships clustered per target groups. BSP MSP Age groups TOTAL 2007 renewed new The distribution of target groups for BSP & MSP is spread among the different ages and different school levels, although the most frequent target is represented by upper secondary schools. The presence of preprimary/primary schools is low and in 2009 only 18 approved partnerships are focused on pupils at 4-7 and 8-11 years old. The cause of this is due both to the lower priority given to the European dimension of education at the pre-primary/primary level and partially - to the impossibility to take part in BSP by this cluster of schools because of the higher minimum age of Bilateral Partnership target group (over 12 years old). Italy is one of the few countries in across European MS in which no national priorities have been/are fixed for Comenius partnerships. 160 In this case the success in reaching specific target groups is not supported by national priorities. In term of geographical representation of the institutional beneficiaries, the Italian situation is very homogeneous. All the Italian regions have always been well represented during the period. Statistics from 2007 and 2009 reveal that the most represented regions are Sicilia, Puglia and Lazio. No deep differences are reported between north and south of Italy, although within the same regions different districts are very active in Comenius Partnership while others are almost non active in BSP & MSP applications. All over the Italian territory the proportion between number of Multilateral and of Bilateral Partnerships is and has remained almost constant during the period To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? The dissemination activity of Comenius Partnerships is considered as a natural action, especially at the end of the Comenius Partnerships. All the projects saved some effort for these activities and the most frequently used tools/media are the following: - Internet websites; 160 Source: 2010 NATIONAL PRIORITIES FOR COMENIUS SCHOOL PARTNERSHIPS IN THE LIFELONG LEARNING PROGRAMME. 185

186 - Expositions, exhibitions; - Studies, reports, other publications; - Photographs; - CD-ROM/DVD; - Seminars, conferences, workshops, other events; - Local TV; - Local Newspaper. In general the most used tools/media referred to are local TV and local newspapers, which provide local and regional dissemination. Cases of national level actions were rarely mentioned by the interviewed beneficiaries. In this sense only the Web tools represent the preferred communication channel to reach a wider audience or larger national territory. Lots of dissemination materials and information about specific Comenius Partnerships are available online (mainly through the institutional beneficiaries websites) although some websites are very simple if compared with the Web state of the art. In some best cases the experience of a common internet blog, accessible from the different partners/ countries of the Comenius Partnership, is a successful tool to support communication and dissemination during all the duration of the Partnership. Concerning the attendees of the dissemination activities and the target groups, the most frequent cases revealed the participation of the whole subjects involved in the beneficiary institution (students, teachers, staff), and: - Education providers in the area of education (the same as the beneficiary); - Enterprises / companies/ firms; - Public authorities. In most cases, detailed by the interviewed beneficiaries, the dissemination events, such as conferences, seminars and TV/Newspaper articles represent the opportunity to keep and maintain the contact between the educational institution (schools) and public local authorities/stakeholders. As regards the LLP NA, the annual reports and the NA s publications always include best cases of successful/interesting projects to be disseminated, because of their specific results, theme or effectiveness. More specifically, Italian Comenius partners can rely on a web platform to store and disseminate the achieved results and the developed products: the European Shared Treasure (EST). 161 EST platform is an infrastructure developed within a project funded under the previous Socrates Programme and fully developed in aimed at collecting the whole set of project results achieved and concrete output created by LLP projects. Developed by a small set of NAs in Europe, it is now a common standard tool to find details about past projects and their results: links and physical documents are listed in the EST platform in order to allow free access to results/products or detailed information about the past project experiences. At the moment (May 2010) the platform includes Comenius and Grundtvig and it has a non-operating connection to other Member States databases. Concerning the sustainability of the partnerships after their formal conclusion, all the interviewees paid attention to using/exploiting some kind of outputs or products developed. Moreover some partnerships themes are considered not limited to the formal duration and further activities have been implemented within the school year programme and considered as a follow-up to partnership s activities. Sometimes the students involved in a Comenius Partnership have taken part to school activities, beyond the class time. Some of the beneficiaries have applied or are applying for a new Comenius Partnership, based on a consolidated network of past partners. In conclusion, disseminating results is considered as added value activity, and always horizontally implemented. Interviewees reported a difficulty in achieving vertical dissemination of Comenius Partnership results and in order to influence national educational policies, from a bottom-up perspective. The LLP NA reported that for the enhanced dissemination action, the relation with the national education policies should be better developed. From the institutional beneficiaries point of view, no huge difficulties emerged, although the general initial feeling with the dissemination activities is very common when teachers face to partnership implementation. The last key factor for a successful dissemination of the results is the updating 161 European Shared Treasure (EST) platform ( 186

187 procedures of the websites, databases, platforms, on which any partnership results have been uploaded and stored. The dissemination should be always aligned to the current state of the art in the specific field. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? The first year of introduction of the new LLP programme was a transitory period for the implementation of the management procedures, too. Up to now (May 2010) the situation is stable and most of the initial issues have been addresses and problems solved: this is the NA point of view. Figure Three institutional subjects for Comenius in Italy EC European Commission Agenzia Scuola LLP National Agency MIUR National Auhority Some of the interviewees identify a lack of stability and some difficulties in the management of procedures and communications besides a lack of clear definition of the roles among the three main subjects in the context of LLP projects and in particular in sub-programme Comenius. This consideration is not specifically reported for BSP and MSP but generally about the overall Comenius. The National Authority did not specify if the difficulties are limited to a specific project stage (application, reporting, monitoring) but the operative procedures seems to be a sort of de-legitimating action of their authority, in favour of the LLP National Agency. On the side of institutional beneficiaries, all the interviewees declared that the guidelines and the requirements for the Comenius Partnership management have been clear and written in a comprehensible way. Moreover they reported efficient experiences in the communication with the LLP National Agency, in relation to quick and clear responses to questions submitted. Finally, it is a recurrent situation that the teacher responsible for the Comenius Partnership within the institutional beneficiary is in charge of coordinating also the administrative issues, cooperating with the administrative director. This successful collaboration is mentioned as a key element in good practice for the best management of a Comenius Partnership Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? From the point of view of the NA, in relation to the management of the operative procedures ensured by the NA and its employees, a lack of resources could be detected for the additional activities, such as analysis of the gathered monitoring data for in-depth research. For example, one of the most interesting data to be analysed is the relationship between the increasingly European attitudes of the Italian schools in order to clarify to what extent it is the cause or the effect of Comenius funded projects. 187

188 The position of the institutional beneficiaries is not unique and among the interviewees different impressions have been reported. Some of them consider that the resources for the mobility are not adequate, while others mentioned that some internal cost reduction policies allowed the school to enlarge the participation to more students than expected for the mobility. In general the management of the funding is an issue strictly dependent by the internal organisation of the beneficiary. All the institutional beneficiaries and the National Authority too, confirmed that a problem is outlined in the low valorisation and recognition of the effort spent by the teachers for the activities of the partnerships. In fact, mobility excluded, the huge effort spent in managing and coordinating the partnership is not covered by the funding and any possible solution is only in charge of the head of School/principal. A suggestion raised from the beneficiaries is that the NA directly specifies some guidelines on this issue in order to align the situation, excluding the autonomy of the individual school on this issue. This solution is also proposed by local offices of the NA, remarking that often Principals need a specific norm to follow and the idea of a new cost item in the beneficiary annual budget could be used in compensation of the spent effort on Comenius Partnerships. Other elements emerged in relation to the adequacy of management resources are: - All the beneficiaries seem to avoid the support of external experts in the management phases, more easily in the application phase. - Finally the beneficiaries mentioned the lack of preparation in the management of internal crisis and emerged difficulties (withdrawing partner, non-collaboration, misalignment of the visions, etc) A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? Useful publications and a recent set of 4 books have been published, in Italian, by the NA aiming at providing an interim overview of the status and results of these 3 years of LLP programme. A dedicated book 162 is focused on the Comenius sub-programme with in- depth reports on School Partnerships. During the period , only the 2008 statistics have not been available for the general public, while 2007 and 2009 are accessible as in-depth reports and database. As reported in the previous section, inadequacy of the available resources adversely affects the ability of the NA to use the monitoring data better in specific analysis documentation and in-depth research. The publications of the NA for the general public include a lot of information gathered from the beneficiaries and collected together by the NA, revealing a bottom-up reception of the relevant issues from the field, confirmed also by the interviewed beneficiaries. The National Authority has not answered this point, deciding to pay more attention to other issues. Finally, cited by the NA, one of the best practices in the field of utilisation of monitoring information is the M- TOOL Project. The Dissemination Project named M-TOOL: Constructing an evaluation tool for monitoring visits, was approved by the European Commission under the restricted call for proposals EAC/13/06. Developed and tested by 6 NAs, including Italy, it is now fully operational and used as an effective tool for monitoring Comenius visits. 162 LLP National Agency (2009), LLP come va? Anno 2009: riflessioni di metà percorso Comenius, Erasmus, Grundtvig e Visite di Studio - Comenius: impronte di cambiamento nella scuola., Florence. 188

189 12. Comenius Assistantship : Bulgaria 1. INTRODUCTION The overall objective of the case study is to investigate the functioning of Comenius Assistantship action in Bulgaria in order to carry out interim evaluation of relevance, effectiveness and efficiency of the Lifelong Learning Programme. The findings are based on desk-research approach and field work realised through conducting of 9 interviews with the National Agency Human Research Development Centre, the National Agency for Vocational Education and Training and beneficiaries from this action. The titles of the documents analysed during the desk-research process are listed up. The list of interviews is presented in Annex 4. The desk-research and interviews were carried out during the period of 26 April to 25 May RELEVANCE The operational objectives of the Comenius sub-programme are focused on two very important criteria: improvement of the quality of education and increasing the volume of mobility. The objective of the Comenius Assistantship action is based on these premises - to give future teachers the opportunity to gain a better understanding of the European dimension to teaching and learning, to enhance their knowledge of foreign languages, of other European countries and their education systems and to improve their teaching skills To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the national policy priorities? What are the links between the LLP and national policies? The Comenius programme (including all its actions) is to a great extent complementary to the efforts of the Bulgarian Government to improve the quality of school education and to make schooling attractive to students. One of the main objectives defined in the School education and pre-school training and preparation development national programme ( ) 163 and its action plan are to ensure sustainability of the process of improvement of teachers qualification and students achievements. Beside this, the Programme of the Ministry of Education, Youth and Science has declared that Bulgaria has to achieve the European level of quality of education. Programmes such as the Lifelong Learning Programme and especially Comenius 164 are one of main sources for fostering such processes. In this respect, the Comenius Assistantship action is perceived as an important instrument for supporting the process of reforming and developing school education in our country in order to respond to the main national political priorities in the field of education. The authorities fully understand the important role the Comenius sub-programme plays in general. Schools and teachers are encouraged and supported to take part in the activities. Participation in the Comenius subprogramme is one of the indicators for measuring teachers' and schools' performance. In order to respond to the main national political priorities in the field of education, national priorities concerning selection of applicants under all Comenius actions, including Assistantship were set in the NA- HRDC Work Programme and these national priorities were strongly complied with in the selection process. For 2010, the national priorities for categories of applicants have been defined as follows: - Students in the last years of their bachelor study or those who are studying at for a Master s degree, - Individuals who gained a pedagogical university degree, but who are not yet employed, - Future teachers in the field of special education/teachers for students with special educational needs, - Future teachers in the field of civil education, - Future teachers in the field of ICT, - Future teachers in the field of foreign languages. 163 School education and pre-school training and preparation development national programme ( ) p Programme of the Ministry of Education, Youth and Science ( ) p

190 The conclusion is that the objectives of Comenius Assistantship action are highly pertinent to the national policy priorities and that they correspond to the objectives set out in the 2007 NA Work Programme. The expansion of this activity to potential teachers provides: - excellent additional opportunity for improving the quality of school education, - opportunity to acquire and upgrade competences in areas of foreign languages 165, - opportunity to acquire competencies in the area of new teaching/learning technologies, assessment of students achievements, etc. The Bulgarian Lifelong Learning Strategy 166 pays special attention to LLP and the possibilities that are provided for effective utilising of training opportunities within the Integrated Programme for LLL of the EU and other international programmes for: establishing conditions stimulating acquisition and upgrading of professional and teaching qualifications; supporting international cooperation and exchange of experience among teaching staff; implementation of innovations at school level practice; facilitation of the process of implementation of new methodologies and methods of teaching/learning; encouraging the integration of students with special educational needs; introducing equal opportunities for learning; development of education in Bulgaria on the basis of implementation of new ICT and improvement of foreign language proficiency To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the lifelong learning needs of the target groups of the programme? Lifelong learning is, for the EU, "the guiding principle for the development of education and training policy". While promoting social inclusion and personal fulfilment, lifelong learning develops people's employability and adaptability, and is therefore a core element of the Jobs and Growth strategy. Lifelong learning includes learning for personal, civic and social as well as for employment-related purposes. It takes place in and outside the formal education and training systems. Lifelong learning implies raising investment in people and knowledge; promoting the acquisition of basic skills, including digital literacy; and broadening opportunities for innovative, more flexible forms of learning. Two main beneficiaries are in the field students and schools. Universities have possibilities to get some benefits indirectly from their students who participated in Comenius Assistantship. The Comenius assistantship remains highly pertinent to the needs of schools to get well qualified foreign language teachers. For many years, the lack of teachers in foreign languages has been a strong negative factor in the Bulgarian educational system. Many initiatives have been adopted in order to improve this situation, Comenius assistantship, however, is not limited in scope and is open for all future teachers. The action can provide sustainability in the process of improving the foreign language skills of future teachers and opportunities for the dissemination of good practice. Interviews with individuals have pointed out some main benefits: first, improvement of language proficiency, and of communication and team work skills: assistant, mentor and school staff all tend to gain in selfconfidence. All these benefits are related to the acquiring of key competencies for lifelong learning. The following target groups were given priority ( ), as indicated in the NA Work Programme: - future teachers of children with special educational needs, - future primary school teachers, - future teachers of information technologies, - future teachers of foreign languages. The majority of applicants have been future teachers of foreign languages. Very few applicants have submitted proposals for assistantship in other subjects, for example History, Geography, Arts, Music, Civil education and etc Language competencies are key competencies for LLL 166 National Strategy for Lifelong Learning for the Period , Republic of Bulgaria, Adopted by Protocol N 42 of a meeting of the Council of Ministers on 30 October 2008, p Number of applications approved by subject for 2008 are as follows: Foreign Language -16 ; Mother Tongue 1; History 1; Music -1; Arts and crafts 2. (Information provided by NA-HRDC) 190

191 All individual beneficiaries who were interviewed confirmed that the action remained pertinent to their individual needs. They confirmed, that objectives set by them in order to carry out this assistance activity, were fully achieved. 168 In 2008 a good balance concerning the educational level of the schools, which applied to host an assistant, was achieved. The highest percentage belonged to secondary schools 50%. Kindergartens were ranked second with 33,34%, followed by elementary schools, 8,33%, and vocational schools, 8,33%. We have observed increased interest in kindergartens in this action under the Comenius Programme. 169 Schools receive information for good European practices and have the opportunity to adapt and implement such experience within different methodological areas. The Universities have an opportunity to extend their cooperation with schools at international level and also an opportunity to develop new partnerships in the future. A very important suggestion has been identified during the interviews with the national stakeholders concerning the possibility of certification of the Comenius Assistantship. Acquiring of a certificate applicable to the requirements for prior learning assessment would markedly increase interest in participation in the activity. The certificate would contribute to the upgrading of the qualification level of future teachers and would also motivate these future teachers to be more aware of how they carry out the Comenius Assistantship. 170 The main conclusion in order to obtain sustainability of the process and create possibilities for lifelong learning is that the LLP ensures benefits for one and the same beneficiary from different sub-programmes and activities through the provision of the widest possible spectrum of professional development opportunities - in order to provide continuation of the process of learning What is the European added value? Would other national schemes / instruments provide enough support to the activities funded by the evaluated LLP sub-programme / action, if there was no LLP? Would adequate attention be given to the challenges faced by the disadvantaged citizens in meeting the lifelong learning needs? European added value is high. All beneficiaries confirmed that it is not possible to participate in such activities if financing is not provided by Comenius action. Universities are not able to provide any subsidies for such activities, nor can students parents be expected to fund them. It is envisaged that the period of experience as a Comenius Assistant will add value, make a difference and have an impact on beneficiaries, their skills, opportunities, and understanding of multicultural aspects of social communication as well as their future employability. Beneficiaries from universities and from school settings, supported by their experience of Comenius Assistantship, will have the opportunity to benefit from their experience in other countries and in different school environments in achieving their successful transition from school to work. In Bulgaria, the last 15 years showed a tendency to lack of motivation among students who gained a degree / a higher education qualification in the field of foreign language to continue to work as teachers. Participation in such activities as Comenius Assistantship motivated them and they found an individual/personal way to understand that they will actually enjoy working as teachers. All interviewed beneficiaries considered that the assistantship programme opened real possibilities for them for better start to their careers as teachers in relation to new teaching standards, set by the Ministry of Education, Youth and Science. Bearing in mind the relatively low number of assistants and host schools, the results of the action will have limited impact: mainly on the assistants and schools concerned. Certainly the assistantship is beneficial to the particular participants and the results could be transferred to other beneficiaries and stakeholders. HRDC tries to intensify this process by inviting assistants and host schools to share experience regarding different Comenius events. 168 The students rated the achievements of personal objectives with range of 4 and 5 out of 5 (the highest) 169 NA report 170 The issue is related to the credit system and recognition of qualification and can be a reliable instrument for measurement of the level of achievements of the results from this action and from LLP, respectively 191

192 The long-term impact and benefit to the beneficiary target groups will need to be evaluated over a longer period. However at this stage, the overall inputs, outputs and outcomes have been an effective way of addressing and successfully meeting the needs for all beneficiaries and achieving assistance to students from different universities. Participants have been given alternative opportunities to learn new methodologies and practical approaches for teaching. They have been introduced to an alternative educational pathway through a learning process and an in-service experience. Many participants have developed alternative learning methods, approaches and tools and put these into practice in their schools, through a combination of the learning process (transfer of competences, awareness of capabilities and skills) with a real opportunity of work experience. In this way they have helped in the process of achieving a transformation of educational practice in schools from traditional education into the more flexible and individually oriented education required in the near future What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? There is no risk of double funding or duplication, because of the fact that NA-HRDC established contracting and reporting procedures, which contribute to avoiding any such negative results. The synergy is ensured between the Comenius action and implementation of National Strategy for LLL, National Strategy for CVT and Implementation of Human Resource Development Programme. A regulatory framework for new opportunities and access to LLL guarantees the synergy of the activities and measures, as for example related to enhancing participation in the EU programmes in the field of LLL, putting an emphasis on studying European languages and on using specialised terminology during the basic and continued training of teachers and etc. 171 The Human Resource Development Centre was established initially with the idea to provide synergy within the European Commission s programmes and activities. Before and now, it has been providing adequate coherence of all activities, measures and programmes. The centre is carrying out accompanying activities to facilitate the implementation of European philosophy of all programmes and activities. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the programme / sub-programme / action could be achieved Effectiveness in Achieving Targets and Contributing to Priorities What is the progress in achieving action / sub-programme targets against financial progress of the programme? What is the extent to which the beneficiaries are satisfied with the outputs and results of different sub-programmes and actions? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? Appropriate funding and human resources for management and implementation of the Comenius Assistantship are fully available at national level. Progress towards achieving the Comenius Assistantship targets in Bulgaria over the period indicates average to high effectiveness of the action s implementation: - First, the funds of the action have been used up to 85%. 172 The number of approved applicants has been stable during the reported time period; although the number of signed contracts for 2009 was lower (11 contracts were signed). From 27 submitted applications in 2009, 18 were approved with the score of 60 points. For all these applicants the appropriate school was identified based on their first request. 171 National Strategy for LLL for the period p NA-HRDC was planned 20 Comenius Assistantships per year for the period In 2007, 18 applications were approved, 16 were completed; in 2008, 21applications were approved, 16 were completed; in 2009, 18 applications were approved, 11 were completed. 192

193 - Second, based on the reports and interviews, the participants were fully satisfied with the Comenius Assistantships. 173 The reliable indicator for measuring the levels of achievement of outputs of the Comenius Assistantship activities is the number of applications received and their quality (concerning the completion of formal requirements and concerning the quality of proposals), which is also and indicator for relevance of the action. One of NA - HRDC observations in relation to proposals under Comenius submitted in 2008 was a considerable decrease of the number of ineligible proposals comparing with the previous years. Another indicator used to measure impact of the Comenius Assistantship activities is the improved quality of final reports from beneficiaries and the decrease in the number of problematic cases. The conclusion is that there is enough capacity of beneficiaries to use the Comenius Assistantship action. 174 During the time of realization of a Comenius Assistantship action of LLP, different cases appeared and were observed, related to mobility barriers: - First case: Application was submitted in due time and in an appropriate template, the applicant was approved by the NA, but later the approved person took the decision not to participate in such activity. - Second case: Application was submitted in due time and in an appropriate template, the applicant was approved by the NA and the assistantship was started. Later the participant decided to discontinue his/her participation. These are still very rare and isolated cases. The reasons for cancelation of Comenius Assistantship in these cases were explained by participants with family or personal issues. There was no indication about problems with the implementation of the activity. - Third case: Some of the participants have applied for extension of Comenius Assistantship. Trying to improve the management an organisational aspect - of decentralised actions, the NA introduced new and updated procedures and documents such as: - Procedure for archiving project dossiers; - Performance assessment form for applicants from external assessors; - Internal rules for project audit; - Procedure for assessment of project reports. NA-HRDC significantly increased activities relating to monitoring and consulting beneficiaries. There is a unit within HRDC, which is responsible for these activities. The results at the individual level are mostly related to the development and improvement of the competencies of the participants as: - student- focused approach to training; - language proficiency; - country history; - specific methods for training of students from primary schools; - working with students with deviant behaviour, - organisation of extra-curricular activities, based on the interviews and reports results achieved. The number of school beneficiaries under the Comenius Assistantship is still very limited. The main reason is that NA-HRDC was not able to appoint foreign assistants who satisfied the school requirements. Apart from this, the results at institutional level are achieved and are related to strengthening the partnership between schools at international level, modernisation of the organization and provision of appropriate teaching and learning processes. There is one more very important result for universities as indirect beneficiaries: the universities got back their students to study for an MA degree with an enhanced level of language proficiency and teaching/training methodology. Through Comenius Assistantship universities have the opportunity to strengthen the university s international network All interviewed beneficiaries and revised reports indicated a score of 4 and 5 for the level of results achievements. 174 Interviews with Krasimira Karasenova and Marchela Miteva from Bulgarian NA-HRDC 175 The interviewed students are now continuing their studies for MA degree. 193

194 What are the main external factors affecting the outputs and results of the sub-programme / action? The main external factors are, first, economic factors. For more than 15 years the Bulgarian financial system has been operated in by a Currency Board. The New Bulgarian Government took measures to increase teachers wages, but there are still not adequate levels of payment, in comparison to those in other European countries. This is the main reason why after graduation the future teachers are not willing to work as teachers. Second, NA-HRDC has listed some main concerns and difficulties related to mobility, such as: - Difficulties with organization of mobility; - Difficulties concerning the communication with the host school prior to mobility; - Difficulties to find an assistant for them, this is a problem especially for the schools in smaller centres of population; - Difficulties in the process of adaptation of the curriculum to the activities of the assistant; - A lack of correspondence between the host school s needs and the assistant s needs. The other external factors, such as differences between national education and training systems, critical factors such as immigration legislation and significant differences between national education and training are not decisive in Bulgaria. HA-HRDC concluded that external factors would be unlikely to affect the implementation of LLP in the next programming period. Bulgarian beneficiaries have been and are demonstrating a permanent interest in co-operation and collaboration within the LLP sub-programmes and activities Effectiveness in Implementing Horizontal Policies How does the sub-programme / action contribute to the horizontal priority of multilingualism? This action contributes directly to the horizontal policy of multilingualism. Comenius Assistantship activities are directly related to further development, upgrading and improvement of language proficiency and skills. All interviewed students confirmed that they have upgraded their language skills in a minimum one foreign language. Some of them have made improvement in more than one language, Italian and English, for example. Participants have had the opportunity to join classes in German, Spanish, English and even Chinese. The NA confirmed that it has made an attempt to give priority to the applicants who apply for a language, which is not widespread. In the selection process, attention was paid to balance in relation to assistantship in different languages, geographical distribution, and gender issue. During the selection phase, the most important criteria taken into consideration were mainly to give a chance to the participants who chose to conduct their assistantship in the field of less popular languages of the EU. The number of applicants who have applied to teach in one of the less popular languages in Europe is still small. The most popular languages chosen to be taught by the applicants are English, French and German. Two applicants chose Spanish and one Swedish. Concerning the subjects, the biggest percentage of applicants submitted proposals to teach foreign languages, civil education, and arts and crafts Is the implementation of equal opportunities (EO) satisfactory in the decentralised actions of the programme? In Bulgaria teachers are mostly female. The students of the faculty of teacher training are mostly female. Only 4 out of 35 applicants for assistantship under Comenius in 2008 were males. In percentage (11%), the balance is not very good, but, taking into consideration the general trends, according to which the number of males in the teaching profession is very low, the balance can be considered good. In 2008, no applications from persons with special needs were received. No special measure to prioritise a gender equality issue is implemented under the Comenius Assistantship. No special criteria and procedures have been implemented for motivation and promotion of Comenius 194

195 Assistantship among students with special needs or disabilities. However, in the case of such an applicant being approved, NA-HRDC would be both willing and able to provide higher financial support to such a participant in the Comenius Assistantship Effectiveness in integrating previous activities into the LLP: Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? The content-related benefits have been implemented through the transfer of Erasmus SMS students/participants to the Comenius Assistantship action. More of the participants in Comenius Assistantship were initially included in Erasmus activities and were willing to build on their positive experiences of Erasmus, improving their competencies further by carrying out a specialised assistantship, for example through upgrading their language proficiency, etc. For school beneficiaries, upgrading the school partnership is a good basis for another mobility activity under Leonardo sub-programmes, providing a possibility for wider range of persons (students and teachers) from the school to be involved. The management-related benefits are implemented through the possibility for the national authorities to integrate all stakeholders in the process of lifelong learning. Such integration gives a possibility to integrate the efforts of all key players and to reflect the European dimension into the national education policy and its implementation. LLP and its sub-programmes and activities guarantee clear differentiation of the roles and responsibilities of all stakeholders in the process of realization of the different components of the national educational policy and integration of good European practices and innovative experience in education and training in Bulgaria. For strengthening the management of LLP, the representatives from the NA concluded that it would be better if the Commission paid attention to the expected results from every sub-programme or activities, rather than use a financial criterion to assess the achievements of the results. The NA has not noticed a substantial difference concerning procedures and administration after the integration of several programmes in LLP. In order to facilitate the administration, NA-HRDC has established a new structure to facilitate further success in the implementation of the LLP at national and international levels. An important suggestion was provided by the NA related to the criteria for assessment of success in the process of implementation of the LLP: the relationship between the expected results of an action and the results finally achieved has to be the main criterion for conclusions. This proposal was made in order to further simplify the application and reporting documents Effectiveness in Reaching Target Groups and Disseminating Results How successful is the LLP in reaching the target groups and institutional beneficiaries of subprogrammes? The Comenius Assistantship is successful in reaching target groups even though the number of school beneficiaries was only 4 in In 2007 and 2008, NA-HRDC worked in close cooperation with universities to select the appropriate applicants for Comenius Assistantship. In the last year, the applicants submitted their applications to NA without pre-selection at university level. The second approach is more useful, because it fully ignores different interpretations of the rules for selection of applicants and provides a transparency in the selection procedures. Dissemination of information about Comenius Assistantship activities is provided through the Internet, the organisation of contact seminars, etc. The participants, who are continuing their studies for MA degrees, disseminate the information and the results among their colleagues at the university level. The participants, who work and study at one and the same time, are disseminating their experience to their colleagues both at the university and at the school/centre level NA-HRDC sources 177 Two of the interviewed persons, Svetlana Mihailova and Svetlana Dimitrova, work as part time teachers at school and training centre 195

196 The fact that in 2008 proposals have been submitted by schools and teachers who have not applied for Comenius grants so far is also a positive sign about the effectiveness of the activities for dissemination of information and good practice. It can be expected that the numbers of applications will be increased based on examples of good practices shared within the community of professionals of teachers in foreign languages To what extent are the Programme results adequately disseminated and exploited? In order to increase the quality of the activities supported under the Comenius programme, NA-HRDC organises annually Comenius information days at which special attention is paid to the foreign language teaching community. The NA supports the collaboration of future teachers with the Network of Innovative Teachers. In May 2009, a Balkan contact seminar was organised with the participation of the Bulgarian, Turkish, Macedonian and Romanian National Agencies and the Comenius sub-programme was presented in particular. The first Comenius Valorisation Conference in Bulgaria was carried out. During the Valorisation Conference, teachers, school managers and representatives from Regional Educational inspectorates and Municipalities took part. The goals of the conference were to evaluate the results achieved, to award certificates to projects of the best quality, to disseminate good practice and to introduce policy-makers and decision-makers to the successful stories of Comenius. The participants of the Comenius Assistantship action took part in such activities in countries of where they conducted their practice, such as the meeting organised by Italian NA in Florence on A book for Comenius Assistantships entitled A Good Practice Guide for Host Schools and Assistants (in English and Bulgarian) is available on the NA web page. At university level the information is available through the Internet sites of the universities. During the interviews, the individual beneficiaries confirmed that they did a presentation after completion of the mobility for their colleagues and that they implemented a lot of new methods learned through their assistantship experience in their daily work. Dissemination of the project results was organised during and after the completion of the assistantship during the practice with the support of mentors to the pedagogical and teacher training departments of local universities in both, native and host countries. The innovative practices and project results have been disseminated and used amongst key stakeholders on a national level through meetings and contact seminars. Beneficiaries organised presentations and these activities are very useful, because they provide practical guidelines to future participants. 4. EFFICIENCY (1-2 pages) 4.1. Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? At the interviews the beneficiaries confirmed that the clarity of requirements for applicants, the application procedure, the clarity of procedures pertaining to financial management and the clarity of the project s reporting requirements have been fully sufficient. The NA also confirmed that requirements for the management, control and monitoring have been clearly defined within the guidelines of NA and that the Commission has fulfilled its guiding role to a high level. The Bulgarian National Agency has developed and is implementing several procedures to maintain a high quality process of launching, assessment of applications, monitoring of assistantship and reporting. At the application stage, quality assurance criteria have been implemented in order to assess the quality of mobility, benefit and relevance. Additional scores are possible if the application is related to national priorities. The national priorities for development of proposals are provided by NA-HRDC. There were some administrative problems concerning applications: some of the applications could not be approved as eligible only because the applicant had forgotten to sign it. In 2007, when the universities were responsible for suggesting eligible applicants, there was a misunderstanding regarding the appropriate language of preparing the application: it was requested that the applications to be submitted in the language of the Comenius Assistantship activities, while the universities announced the requirement for the application to be prepared 178 Sources: NA and participants reports. 196

197 in the language of the host country. Such problems did not appear in 2009, because NA was disseminating the information about the rules and procedures correctly. At the implementation stage, the responsibility has been divided between participants and the NA. In the case of cancellation or request for continuation of the assistantship, the NA was authorised to make a decision only after interim monitoring of the performed tasks and achieved results. The participants became familiar with all management procedures before contracting with the NA Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? National stakeholders possess adequate resources to ensure sound management of the programme through clearly defined and transparent procedures and dissemination of the results. All documents are available on the Internet site of the NA and a lot of information can be found on the universities websites. As regards the Comenius Assistantship action, there are no shortages of management resources. No functions are fulfilled through external contractors for this action Is the monitoring information used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? The data collected during the monitoring process is not used for any other purpose except reporting to the EC. The Comenius Assistantship activity is reported within the documents related to the implementation of the European dimension of educational policy, for example, universities annual reports, etc. 197

198 13. Leonardo da Vinci Professionals in Vocational Education and Training (VETPRO): the Netherlands 1. INTRODUCTION This case study elaborates on the Leonardo da Vinci Professionals in Vocational Education and Training (VETPRO) in the Netherlands. This action of the Leonardo da Vinci sectoral programme aims at the support of transnational mobility of persons responsible for vocational training and/or human resources. An overall purpose of this case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme action. 2. RELEVANCE The main objective of this section is to describe the European added value of the selected action To what extent the intermediate and specific objectives of the action remain pertinent to the national policy priorities? The contribution of the VETPRO action to the national priorities for VET is satisfactory. On institutional level, the relevance of the VETPRO is felt even more. The Leonardo da Vinci in general, is considered to be highly pertinent to national policy priorities. Internationalisation Agenda VET Internationalisation within VET receives more and more attention given the international orientation of VET in the Netherlands and the objectives of the Lisbon and Copenhagen Agenda. In the Strategic Agenda Internationaliseringsagenda MBO (Internationalisation Agenda VET) 179 the Ministry has set short-term and middle-long term objectives: - Stimulate inbound and outbound mobility, by making use of national and European programmes; - Continuing cooperation in Europe (focus on the use of European instruments); - Strengthening the image of the Dutch VET. The instruments to pursue these aims are: - Better embedding of the international orientation of VET students in the qualification structure of VET; - Optimisation of the use of the Leonardo da Vinci programme; - Stimulate transparency and comparability of vocational qualifications with Europass, the implementation of EQF (eventually ECVET) and the Acknowledgement of prior (experiential) learning (APL); - Improving communication on the representation of the Dutch Vet sector within an international context. Responsibilities The educational sector in the Netherlands is organised in a much decentralised way: many responsibilities are delegated to the institutions. This accounts as well for the topic of internationalisation. On an institutional level staff mobility is seen as a means for quality improvement within the institute. For instance, Deltion College has draw up its ambitions with regard to internationalisation, stating that from 2011 onwards, 10% of the staff needs to be abroad for at least 5 working days (within the VETPRO action). Objectives VETPRO action The Leonardo da Vinci programme plays an important role in realising the Dutch strategic Agenda and is mentioned explicitly as instrument. The action Mobility for Professionals in Vocational Education and Training (VETPRO) of the Leonardo da Vinci sectoral programme aims at the support of transnational mobility 179 Ministry Education, Culture and Science, Internationaliseringsagenda MBO 198

199 of persons responsible for vocational training and/or human resources. It therefore is of utmost importance within the light of the strategic agenda for the VET sector. The general objectives of this mobility action within Leonardo da Vinci sectoral programme are: - To support participants in training and further training activities in the acquisition and the use of knowledge, skills and qualifications to facilitate personal development in a professional context. - To support improvements in quality and innovation in vocational education and training systems, institutions and practices. VETPRO is seen as a drive wheel for internationalisation in the VET-college: student mobility will be stimulated further, when the staff is enthused about international experience To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? The interviews show that the VETRO action and its objectives remain highly pertinent to the needs of the target groups. The amount of funds available and the absorption of funds for this action are stable over the years. This makes that the number of mobilities is stable as well. There are more applications than that there is funding available. Despite that the satisfaction is not measured in a systematic way, reflections on the action are communicated to the National Agency on a regular basis by the officials in the institutions responsible for the VETPRO action. The target groups of the VETPRO action are: professionals in vocational education and training (such as teachers, trainers, vocational training staff, guidance counsellors, those responsible for training establishments, for training planning, occupational guidance within enterprises, human resource managers in enterprises). The application for funds within the VETPRO-action is coordinated by the VET-college (ROC). Most of the time, the application focuses on a specific target group, for instance, teachers within a specific field of practice, counsellors and supporting staff. The mobility can endure 1 to 6 weeks. From interviews it became clear that the VETPRO action lives up to the lifelong learning needs of the participants. There is sufficient room to create intensive learning environments to stimulate international, intercultural exchange. However, the interviewees pointed out that possibilities within the VETPRO action should be enlarged to enable staff to visit students abroad. Combining staff and student mobility might be an interesting way to increase the quality of mobility. The VETPRO action s most pertinent effect can be seen on individual level: participants of the action report a positive development on their personal development, acquisition of intercultural skills and improvement of their professional practice. Secondly, the impact is felt on institutional level: by improving the international quality of staff, the quality of provision and the attitude towards the international dimension within the VET institutes improves The European added value. Would other schemes / instruments provide enough support to VETPRO related activities, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? European added value (EVA) is found in actions that cannot be sufficiently undertaken at Member State level, and therefore, for reasons of scale or effects, are better undertaken by the EU. Added value can be understood as the value resulting from an EU intervention that is additional to the value that would have resulted from intervention at national or regional level. The European added value of the VETPRO action is substantial. Without the VETPRO action almost all staff mobility would not take place in the Netherlands. The vocational education & training (VET) sector in the Netherlands knows little possibilities for international cooperation. Most of the work in this field is supported by the LLP. There are a lot of institutions very active in international cooperation and exchange and the LLP is the main vehicle for this. The Leonardo da Vinci subprogramme is an important element in the strategic choices of the VET institutions. Beneficiaries indicate the importance of the student grant since students in VET need an incentive to go abroad. Furthermore, the funding is essential to help in the costs made in going abroad. 199

200 Despite that internationalisation is a priority topic, on a national level there are not many supporting schemes to stimulate staff mobility. The Leonardo da Vinci is responsible for more than 97% of the mobility within programmes of lecturers in The national BAND programme is responsible for the other 3%. The percentage of lecturers travelled abroad within the Leonardo da Vinci programme is approximately 3%. 180 Another programme that ran until recently (last round in 2009) was Programme Internationalisation VET.F 181 F This programme was based on co-funding and aimed at increasing the international knowledge (of languages) and skills of students and staff in VET (at secondary and tertiary level). Despite these initiatives, the Leonardo da Vinci sub programme funds activities that cannot be funded within other programmes. Mainly on individual and institutional level an omission will be felt when VETPRO would no longer exist. This not only in relation to the direct exchange of staff, but also within the broader scope of establishing networks for future transnational cooperation, student mobility and the set up of future projects What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? The National Agency in the Netherlands consists of three separate organisations, each responsible for a specific educational sub sector. For the VET sub sector, CINOP is the responsible agency. CINOP operates in the vicinity of the VET-sub sector and coordinates other internationalisation initiatives for VET as well. This makes that at sub-sector level the preconditions for coherency of efforts are established within the National Agency s VET-organisation, CINOP. As a negative result of the separated organisation of the LLP however, cooperation on sub programme level is hindered. Despite improvements can be seen in recent years, synergies between- and structural coordination with the different sub-programmes is non-existent. One can question, what would be advantages for synergy between the three organisations in relation to VETPRO. Each sub sector has its own staff mobility scheme, having its own specific target group and benefitting institutions (Erasmus: higher education institutes, Leonardo da Vinci: VET-colleges, Comenius: primary and secondary schools, Grundtvig: adult education institutes). However, taking a closer look there is an area where cooperation and a joint approach might be helpful, namely in the field of adult education. VET-colleges in the Netherlands are responsible for a big part of (formal) adult education, but this fact receives little attention in the Leonardo da Vinci programme. Cooperation between Leonardo da Vinci and Grundtvig could strengthen the adult learning dimension in VET-colleges. And since VETPRO is for all VETrelated staff, staff related to adult learning could use VETPRO as well. Within the National Agency this has not yet been an issue. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and the results of the action? What explains the match between the needs of beneficiaries and their satisfaction with the outputs and results? The VETPRO-action is highly effective in reaching its targets. In achieving this, the VETPRO-action absorbs all it funds. Furthermore, the National Agency has given priority to the VETPRO-action in case shortages occur in the funding of the other actions within the Leonardo da Vinci programme. The VETPRO action covers only a 180 Nuffic (2009), Internationalization Monitor of Education in the Netherlands Programma Internationalisering Beroepsonderwijs (PIB): Hhttp:// 200

201 part of the total budget for the mobility actions. Since, the institutions applying for funds only receive a part of the requested funding; the National Agency is able to accept almost all applications. This varies between funding 100% and 25%. The achievement ratio s are high as well, exceeded the targeted number of exchanges in 2007 and 2009 (Table ). Table Achievement ratio s VETPRO in Target Achieved Achievem Achievem Achievem Target Achieved Target Achieved ent ratio ent ratio ent ratio VETPRO % % % Source: Work programmes and Yearly reports, calculation Research voor Beleid The achieved numbers are strongly related to the available funding and the nature of the mobility: whether the mobility action endures 1 or 2 weeks (both count as one mobility) (see table ). Table Duration of mobility within VETPRO-action in Duration age 1 week 91% 2 weeks 9% Total 100% Source: Annual report 2009 The minimal duration of one week is considered problematic by applicants, since it is difficult to find replacement of VET professionals for that time. 182 Participants indicate that it is difficult to get staff enthused to travel abroad for a whole week and deem it unnecessary for the professional and personal development. By better preparation and a more intensified programme, the duration of mobility can be shortened. As a consequence, more staff member will be able to participate both in personal terms and financial terms (i.e. with the same amount of money, more professionals can have an international experience). With regard the satisfaction of the mobility action of VET-professionals, the interviews and reports indicate that participants are highly satisfied with the VETPRO-action. The impact of the VETPRO-action is firstly felt on an individual level as an enrichment of the professional practice and an incentive for personal development. Secondly, on an institutional level, the impact is felt in broadening and strengthening the internationalisation agenda of the VET-college and the impact on quality improvements in the college. Furthermore, beneficiaries indicate that VETPRO is essential in setting up and developing networks and finding partner organisations to cooperate in the future. For instance both Deltion College and the umbrella organisation (MBO-raad) indicate that staff mobility is essential and effective in enlarging the awareness of internationalisation and stimulation of students to go abroad. Impact on a system level is highly ambitious to expect from this action. Moreover, the objectives of the VETPRO action (supporting participants in training and supporting quality improvement) are more appropriate for the institutional level than the system level. Beneficiaries put emphasis the importance of VETPRO: without VETPRO, internationalisation within the VETsector would become very difficult. The other Leonardo da Vinci actions depend to a large extent on the contacts established by the VETPRO-action What are the main external factors affecting the outputs and results of the action? The increased attention on policy level as well as on institutional level, towards internationalisation in VET, international entrepreneurship, international business and globalisation affect positively the outputs and results of the VETPRO-action on the long run. Within the institutions, there is often a positive policy towards stimulating staff to go abroad. The mobility of staff is seen as drive-wheel for internationalisation in the institutions. All in all, the interviews indicate that internationalisation within VET is a challenge for 182 See Yearly report 2007 and

202 institutions as well as individuals. It demands commitment of both the institute as the participants to overcome the barriers described here below. External factors that negatively affect the outputs can be seen on system, institutional and individual level. - System level: 1) An external factor is the economic conjuncture: the attention towards internationalisation in VET is sensitive to the economic developments. Internationalisation is not (yet) seen as core activity of most VET-colleges, therefore, when budget-cuts need to made, internationalisation is placed in the danger-zone. 2) A second external factor on system level is the mandatory number of teaching hours. Since, the institutions are mainly accredited and judged on the number of teaching hours (850 hours on a yearly basis) 183, mobility of staff is seen as contraproductive: special arrangements need to be made to enable staff to go abroad and to find replacement. Given the fact that institutions are judged on the number of teaching hours, staff mobility is not regarded a priority on system level. - Institutional level: The Yearly report 2009 mentions the following external factor that negatively affect the outputs and results of the VETPRO-action on an institutional level: it appears to be difficult for participants to realise a placement abroad in combination with the mandatory contact hours a teacher should make. At some VET colleges less priority is given towards internationalisation because of the priority to implement Competent Based Learning and due to re-organisations of VET colleges/institutes. 184 In short, mobility of VET professionals interferes with internal organisation of the VET-colleges. - Individual level: Another external factor that affects negatively the willingness of staff to participate is the difficulty of combining family life and a full week abroad. It appears difficult for staff to go abroad for at least 5 working days, using the weekend for travelling. Some interviewees express the wish to make the duration of mobility shorter, making it more desirable for more staff; others want to hold on to the current duration, to guarantee the impact on the personal development Effectiveness in reaching the target groups and disseminating results How does the action contribute to the horizontal priority of multilingualism? The mobility of VET-professionals attributes to the horizontal priority of multilingualism since the participants are enforced to speak foreign languages (mostly English). The duration of the mobility is however too short to expect linguistic improvements on a large scale at individual level and one can assume that most participants have already sufficient knowledge in English. Therefore, contribution to multilingualism is limited, but satisfactory. No improvements are suggested during the interviews to increase the contribution to the horizontal priority of multilingualism How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? The VETPRO action is not specifically aimed at the horizontal priority of equal opportunities. The selection of participants for a mobility action is based on intrinsic and content-related criteria primarily. In the application of funds it is possible to receive extra funding for people with special needs. According to the data, more men participate in VETPRO than women (respectively 61% and 39%). 185 As selection criteria within participating institutions, the motivation of the professionals is the most important criterion Effectiveness of the integration of the previous activities into the LLP? Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? What are the advantages and disadvantages of the integration of previous programmes into the LLP? The VETPRO-activity, mobility of VET professionals, existed already under the Leonardo da Vinci II programme. In the integration of the different sub-programmes into one Lifelong Learning Programme not Yearly report Yearly report

203 much has changed to the VETPRO-action. The interviewees indicate that VETPRO is used in relation to other Leonardo da Vinci actions (Transfer of Innovation and partnerships), however further integration of these actions is desirable, for instance, using VETPRO to visit students abroad and finding new apprentice-places abroad. No interaction is recorded by the interviewees of coordination of actions across the different subprogrammes (for instance Erasmus, Grundtvig) Effectiveness in reaching the target group and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? The VETPRO action is successful in reaching the target groups and institutional beneficiaries. On a yearly basis all funds are absorbed and more institutions participate (causing decreasing amounts of money per institution). An effort could be made to include other types of institutions. Individuals cannot apply for a VETPRO subsidy directly at the National Agency. Instead, the application needs to be submitted by one of the following organisations: - Institutions or organisations providing learning opportunities in the fields covered by the Leonardo da Vinci sectoral programme; - Associations and representatives of those involved in vocational education and training, including trainees, parents and teachers associations; - Enterprises, social partners and other representatives of working life, including chambers of commerce and other trade organisations; - Bodies providing guidance, counselling and information services relating to any aspect of lifelong learning; - Bodies responsible for systems and policies concerning any aspect of lifelong learning vocational education and training at local, regional and national level; - Research centres and bodies concerned with lifelong learning issues; - Higher education institutions; - Non-profit organisations, voluntary bodies, NGOs. From the data available, it appears that most of the applications come from VET-colleges and consortia of VET-colleges. In the last years, only one organisation was not classified as VET-college: namely Knowledge Centre Trade (Kenniscentrum Handel). As has been indicated before, the organisation coordinating the VETPRO-action, CINOP, operates in the vicinity and in close contact with the VET-colleges. They exchange knowledge and experience on a regular basis, provide information folders and organise information days and workshops. Furthermore, a large share of the VET-colleges is already involved in Leonardo da Vinci projects and VETPRO in particular. In order to reach the institutes, different activities are implemented. Flyers have been developed and distributed; a brochure is published on the website on how to participate in the VETPRO-action and regular contact (phone and ) between the National Agency and the institutions enable stakeholders to exchange views, ideas and information rapidly. Furthermore, workshops have been organised and information has been distributed during information-markets across the Netherlands. The primary focus of the National Agency is on VET colleges, with regard to other organisations, there is still room to expand the reach of the VETPRO action on different target groups To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels 186 )? As has been indicated previously, the impact and results of the VETPRO action can be identified at individual and institutional level. The focus is on the individual development of the VET professional. Because of this individual development, the institute gains skills and competences in international affairs. The effects of this process are difficult to mainstream amongst other organisations. However, in the application procedure the applicants are encouraged to think about potential organisations that would be interested in having knowledge of the results of the VETPRO action. 186 Horizontal mainstreaming refers to the project results and good practices adopted by other stakeholders. Vertical mainstreaming refers to the integration of the good practices and project results into general policy on national and/or European level. 203

204 A key role for dissemination exists for the umbrella organisation for VET-colleges (MBO-raad) and the National Agency (CINOP). A lot of emphasis is placed on the exchange of experiences and the description of good practices. In the Jaarboek Leonardo da Vinci, best practices are presented and during workshops and in information folders, good practices are presented. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? In general, beneficiaries are satisfied with the way the VETPRO-action is implemented. They emphasise that procedures could be simplified, but they are workable for most applicants experienced in the VETPRO-action. Application In the application, the National Agency follows the LLP guidelines. No national priorities have been added. The VET-colleges are experienced in filling in the application format. The advisors of the National Agency can be asked for advice by of phone. As minor flaws in the application procedure the following elements can be mentioned: - In general, it happens that the application-formats are distributed very late and close to the deadline, which causes problems for applicants, when changes are made in the format with regard to previous years. - More continuity in procedures is desirable (not too many changes in application format on a yearly basis). The interviewees indicate that despite that drawing up the application is a time consuming exercise, a lot can be based on previous applications. Changes in the application formats, adding new indicators, new information requests, make the procedure more time consuming. Implementation The implementation of the VETPRO action does not lead to problems. The National Agency is very approachable when difficulties occur. On institutional level more attention could be given to VETPRO: for instance, the mobility of staff could be better prepared (language training, training in dealing with cultural differences, etc.). Furthermore, the duration of the period abroad might be shortened (less than a week). Monitoring and reporting Monitoring is carried out on a regular basis by the National Agency (CINOP). The advisors have close contacts with the VET-colleges and keep updated by regular phone conversations or . The use of the administrative tool LLPLink has not been developed towards its full potential. Most administrators of the National Agency are having difficulties in using it and the system is at this point not very robust and stable. All organisations responsible for their sub programmes (including CINOP) have therefore duplicate book keepings, which are not integrated in one bookkeeping system for the whole LLP. The final reporting of the mobility actions is evaluated by the advisors of CINOP and feedback is provided to the participating institutions Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? The National Agency (CINOP) has at its disposal 11 FTE to carry out the whole Leonardo da Vinci programme. The total budget of the programme in 2009 is almost 10 million. From both the National Authority (Ministry of Education, Culture and Science) and the European Commission CINOP receives half a million to implement the whole programme. 204

205 CINOP is able to provide all the necessary services for implementing the programme. Judging on the experiences from beneficiaries, the coordination and implementation of the VETPRO-action is very adequate and manageable. No shortages of any kind seem to occur in relation to the Leonardo da Vinci programme A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to the National Authority and European Commission? The data collected by the beneficiaries is analyses by the National Agency and used to supervise and support the beneficiaries. Furthermore, the data are used to serve as good practice at several occasions possible (workshops, folders, Yearly NA Reports etc.). 205

206 14. Leonardo da Vinci Transfer of Innovation : The Netherlands 1. INTRODUCTION This case study elaborates on the Leonardo da Vinci Multilateral Projects Transfer of Innovation in the Netherlands. This action aim is to improve the quality and attractiveness of the European VET system by adapting and integrating innovative content or results from previous Leonardo da Vinci Projects, or from other innovative projects into public and/or private vocational training systems and companies at the national, local, regional, or sectoral level. The process for transferring innovative training content or results includes the following: - Identifying and analysing targeted user requirements; - Selecting and analysing innovative content to meet these requirements and analysing the feasibility of transfer; - Integrating (or certifying) it in European, national, regional, local and/or sectoral training systems and practices. An overall purpose of this case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme action. The information presented in this case study is based on the following sources: - In-depth interviews with the National Agency (2 persons), the European Commission (1 person), the MBO raad (1 person), institutional beneficiaries (5 persons). - Desk research, which sought to analyse National VET policies, monitoring data, implementation statistics, project reports, thematic studies and other documents. The interviews and desk research were carried out during the period of April 19 th to May 31 st. 2. RELEVANCE The main objective of this section is to describe the European added value of the Leonardo da Vinci Multilateral Projects Transfer of Innovation To what extent the intermediate and specific objectives of the action remain pertinent to the national policy priorities? The contribution of the Transfer of Innovation action to the national priorities for VET is satisfactory. The Dutch national agenda for VET fits the theme and priorities set on European level, especially those formulated for call 2008 and Besides, the Dutch VET international agenda emphasise the optimisation of the use of the Leonardo da Vinci programme, but also the need to find good working methods outside the Netherlands and test them in Dutch practice. The Dutch VET agenda In the Dutch strategic agenda for Vocational Training and Adult Education five policy aims are identified: (1) improvement of the connection between VET and the labour market; (2) improvement of the quality of educational provision; (3) improvement of the connection within the professional columns; (4) active and sustainable participation of different target groups to education and the labour market; (5) clear positioning of the VET sector. With the help of these five policy aims the Netherlands is working on improving the quality of the VET sector and a better match between supply and demand. In order to reach these aims some pre-conditions should be met, such as the position of educational staff and the appreciation of the profession (see action plan Leerkracht van Nederland). Moreover, legislation should be adapted based on competence based qualifications, and e.g. including an article that allows experimentation in the WEB (Wet Educatie en Beroepsonderwijs). Furthermore, attention should be paid on the financial organisation of the VET sector. Other pre conditions are new governmental relations and scale of the VET sector. In addition to this strategic VET agenda, the Ministry has further elaborated an international agenda for the VET sector, influenced by an increasing international orientation of VET and the objectives of the Lisbon and Copenhagen 206

207 Agenda 187. In the Strategic Agenda the Ministry has set short, middle and long term objectives: (1) Stimulate inbound and outbound mobility, by making use of national and European programmes; (2) Continuing cooperation in Europe (focus on the use of European instruments); and (3) Strengthening the image of the Dutch VET. The instruments to pursue these aims are: (1) Better embedding of the international orientation of VET students in the qualification structure of VET; (2) Optimisation of the use of the Leonardo da Vinci programme; (3) Stimulate transparency and comparability of vocational qualifications with Europass, the implementation of EQF (eventually ECVET) and the Acknowledgement of prior (experiential) learning (APL); and (4) Improving communication on the representation of the Dutch Vet sector within an international context. The link between European priorities and national themes The Leonardo da Vinci programme, and the Transfer of Innovation Action, plays an important role in realising the Dutch strategic Agenda and is mentioned explicitly as instrument in the national strategy. The European priorities as reflected in the call for proposals 200 and 2009 are (1) improving the quality of VET systems and practices, (2) the skills and competences of VET teachers and trainers, (3) transparency and recognition of competences and qualifications, (4) skills development of adults in the labour market, (5) raising competence levels of groups at risk and (6) developing the learning environment. The European priorities, as included in the call of 2007, were more focused, specifically addressing certain (European) tools, such as innovative e-learning content, as well as European Credit transfer for VET (ECVET) and validation of non-formal and informal learning. Analysing the relation between the national and European priorities one see a great overlap. This was also reflected during the interview with the national agency, Cinop, in which was indicated that the European priorities remains pertinent for the Dutch situation, although some more focus in future call for proposals would be welcome. Therefore, among as one of the reasons, the national agency added some national priorities to the European priorities (Leonardo da Vinci Koersbepaling Transfer of Innovation 2007; Europese prioriteiten & Nationale thema s Leonardo da Vinci ToI, 2009). For the call 2007 additional priorities were added on national level, such as early school leavers, profession column, vocational learning in companies, entrepreneurship, technical training courses, EQF, Euroskills, European mobility for people on the labour market, and specific target groups in secondary vocational education. The European priorities formulated for the call for proposals in 2008 and 2009, we more broad, so it was easier to embed them to the national context. Dissemination and exploitation of results is a key feature of TOI projects, as already started during the implementation of the second phase of Leonardo da Vinci. 188 The importance of dissemination of good practice is reflected in a great extent in the general Dutch internationalisation strategy that includes the following priorities: a) internationalisation of educational programmes, b) promoting dissemination of good practices across the whole country, c) continuing of mobility and cooperation, d) boost multiculturalism and e) boost monitoring and evaluation. In general, internationalisation is regarded as a means to safeguard the quality of education To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? The target groups of the ToI action are all those in vocational education and training, other than at tertiary level, as well as the institutions and organisations facilitating such education and training 189. The interviews show that the ToI action and its objectives remain highly pertinent to the needs of the target groups. 187 Ministry Education, Culture and Science, Internationaliseringsagenda MBO 188 In the second phase of Leonardo da Vinci projects were required to present a formal plan for Valorisation (term now being replaced by dissemination, exploitation, mainstreaming & multiplication). Besides, additional funding was made available during the 2nd LDV Phase to fund Valorisation Conferences & specific Transfer of Innovation projects (initially LDV although subsequently extended to Socrates). 189 Such as institutions or organisations providing learning opportunities in the fields covered by the Leonardo da Vinci sectoral programme, associations and representatives of those involved in vocational education and training, including trainees, parents and teachers associations, enterprises, social partners and other representatives of working life, including chambers of commerce, other trade organisations and sectoral organisations, bodies providing guidance, counselling and information services relating to any aspect of lifelong learning, bodies responsible for systems and policies concerning any aspect of lifelong learning vocational education and training at local, regional and national level; Research centres and bodies concerned with lifelong learning issues, organisation at tertiary level (higher education institutions) can participate in the projects, but the results should not target those attending vocational education and training at tertiary level; Non-profit organisations, voluntary bodies, NGOs. 207

208 The priorities as defined in the different call for proposal remains pertinent to the needs of the target groups in the Netherlands, mostly Regional Education Centres (ROC s). One of the beneficiaries (VET college) interviewed, for example stated that the ToI action excellently fit their broader internationalisation strategy in which they indicate that foreign partner institutes and participation in international projects provide the opportunity to renew the educational offer of their learning institute and to take notice of new didactical and methodological views and will contribute to innovation and continual improvement of the educational offer. Besides, as indicated by several interviewees, the priorities as included in call for proposals are sufficiently broad to fit the strategic policy of their organisations. This flexibility is especially important for the organisation of the Dutch VET system which is organised in a decentralised way: many responsibilities are delegated to the VET institutions. In this way the LLP reflects the needs of the target groups by providing them the opportunity to apply for a project based on their own needs. Another indication of the fact that there is a clear demand for the Dutch ToI action is that it receives a high number of applications and is using all the funds available, although the number of application is decreasing in the last two years The importance of paying extra attention to transfer of innovation in a separate action on decentralised level was considered by several interviewees as very important and need more attention in future programming and should also be considered by other sub programmes 190. Developed products, methods, procedures are, even in the current programme, still not fully exploited, especially the project outcomes of the Development of Innovation Action. However, some of the beneficiaries indicated that the National Authority is mainly focused in trying to get innovations / good practices into the Netherlands, instead of transferring innovations to other countries. The organisation of the programme would be helped if there was more a balanced selection of projects that are selling and buying their innovative practices from other countries The European added value. Would other schemes / instruments provide enough support to ToI related activities, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? European added value (EVA) is found in actions that cannot be sufficiently undertaken at Member State level, and therefore, for reasons of scale or effects, are better undertaken by the EU. Added value can be understood as the value resulting from an EU intervention that is additional to the value that would have resulted from intervention at national or regional level. The European added value of the ToI action is substantial. Regarding the general availability of alternative funding, the information collected during this case study points to the conclusion that for the activities carried out within ToI, no alternative sources of funding are available in the Netherland, or to a limited extent. Therefore the international agenda for the VET sector clearly indicates the importance of making better use of the Leonardo da Vinci programme for continuing cooperation in Europe (focus on the use of European instruments). Besides, some respondents clearly see the role of the National Agency, as advising partner of the project, as clear added value, providing advice on the content related issues while implementing the project, not possible without the programme. Finally, respondents indicate that without the ToI action no (sustainable) partnerships would be developed, as a result of the project. In this way the ToI action really fill a gap in the Dutch VET system and is therefore an important element in the strategic choices of the VET institutions. Finally, beneficiaries have mentioned the fact that participation in the LLP and access to EU funding gives their project additional status which makes the implementation significantly less difficult. Thus by being able to show the support provided by the LLP, it is easier for projects to attract additional participants and also to gain additional funding. Thus, the status provided by participation in the LLP can enable projects to exploit their results and eventually continue without the funding. 190 DoI actions were set up as a centralised action, while the ToI was set up as a decentralised action to be able to adapt to national needs and further exploit the outcomes of the LLP. This makes the Leonardo da Vinci sub programme distinct to other sub programmes that also have a lot of MPs but not on a decentralised level. Besides ToI emphasise the increased attention for dissemination & exploitation of project results, as already started during the implementation of the second phase of Leonardo da Vinci

209 2.4. What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? Synergies between the ToI action, the wider Leonardo da Vinci programme and other sub programmes could appear on different levels, such as reaching new target groups / audiences of the Programme, transferring innovations from other educational sub sectors to the VET sector, integration and harmonisation in the procedures, reducing duplication of efforts and administrative simplification. For identifying synergies it is important to take into account the organisation of the Dutch LLP. The National Agency in the Netherlands consists of three separate organisations, each responsible for a specific educational sub sector. For the VET sub sector, CINOP is the responsible agency. CINOP operates in the vicinity of the VET-sub sector and coordinates other internationalisation initiatives for VET as well. This makes that at sub-sector level the preconditions for coherency of efforts are established within the National Agency s VET-organisation, CINOP. As a negative result of the separated organisation of the LLP however, cooperation on sub programme level is hindered. Despite improvements can be seen in recent years, such as harmonisation of procedures and administrative simplification, content related synergies between- and structural coordination with the different sub-programmes is still underdeveloped. Analysing synergies in reaching target groups and transferring innovation between sub sectors one need to be aware that each sub sector has its own challenges, having its own specific target group and benefitting institutions (Erasmus: higher education institutes, Leonardo da Vinci: VET colleges, Comenius: primary and secondary schools, Grundtvig: adult education institutes), although overlap appear between the ToI target groups and other educational sub sectors, such as the field of (formal) adult education for which VET colleges in the Netherlands are responsible for a big part. Cooperation between Leonardo da Vinci and Grundtvig could strengthen the adult learning dimension in VET-colleges. Moreover, changes in the sub programme resulted that tertiary level applicants could not apply for the Leonardo da Vinci programme, while this was the case in the previous Leonardo da Vinci programme. Now they are funded under the Erasmus programmes that do not provide such a decentralised action. During the interview with the NA it appeared that still a high number of higher education institution approach CINOP (NA) with a request to apply for the ToI action. In the previous programme period a high percentage of applicants for Leonardo da Vinci pilot projects (estimated 25% by one of the respondents) consisted of education institutes on tertiary level. Higher-level professional education & training appears to sit between the 2 sub-programmes. Another interesting question is whether innovations taking up in ToI projects are coming from previous Leonardo da Vinci projects or other educational subsectors. During the interviews and analysis of all selected ToI projects over the period , it became clear that less innovative content or project results are coming from other educational sectors. This is mainly caused, by the fact that the VET sector in the Netherlands (BVE sector) is quite distinct form other sectors and according to respondents already far developed compared with other educational sectors, that makes it less interesting introducing innovations from other educational subsectors. As a result most of the innovations are coming from other organisations within the VET sector, often from beneficiaries own networks outside the LLP. Striking is that less innovative content is coming from previous Leonardo da Vinci Projects. The main reason for this is that applicants, but also the NA, are hardly aware of these project(s) (results) within and outside the Netherland. Besides, less Dutch VET organisations are participating in DOI projects, resulting in less knowledge of the innovations developed within this action. Participating in the DOI action is not considered as attractive by Dutch applicants due to the fact that these projects are bigger projects with larger budget, taking a lot of responsibility for the lead partner. Herewith we can conclude that the link between both actions could be improved for the Netherlands. Respondents also indicate that it is sometimes difficult to identify the differences between the ToI and DOI projects. Transfer of Innovation (TOI) projects can and will involve some aspect of development, although normally focused upon adaptation than starting a new. Development of Innovation (DOI) projects will most often involve some aspect of transfer, with very little being totally new in European VET, although any transfer should be less upon existing products or processes and more upon findings and recommendations. The difference can often be a matter of judging the novelty and the balance of development versus transfer activity. The NA also indicates that they are hardly aware of the projects selected in the specific calls of the programme such as for EQF and ECVET. 209

210 Although, the Commission has put an increasing effort in better disseminating and exploiting the outcomes of the LLP, by establishing project databases and other D&E activities, it would be welcomed that potential applicants, but also the NA, are better informed about good practices financed by the LLP programme. Project databases are helpful but not exactly feeding the policy needs of NA and potential beneficiaries. The project databases only provide a summarising overview of LLP projects, lacking analysis indicating what can be learned from the projects. Positive experiences are indentified with the development of thematic monitoring groups, guided by the national agencies, clustering different Leonardo da Vinci projects together, in order to identify successful approaches for further dissemination and exploitation. The NA clearly sees an added value in these activities, but also realise that it cost extra time and money for their organisation. According to several respondents the Commission could play a more active role informing NA on possible good practices that should be further exploited with the help of the LLP. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and the results of the action? What explains the match between the needs of beneficiaries and their satisfaction with the outputs and results? Overall, the progress towards achieving the national ToI targets in the Netherlands over the period allow indicating of an effective implementation. Firstly, the funds of the action are used for 100% each year, however, the number of projects are decreasing each year. Secondly, according to the interviewed beneficiaries, a vast majority were highly satisfied with the project outcomes. Analysing the statistics of Transfer of Innovation action in the Netherlands over the period one see that the targets are achieved for 2008, realising 13 ToI projects with a target of 12 (achievement ratio of 108%), while targets were not reached for 2008 (10 out of a target of 12: 83%) and 2009 (8 out of a target of 12: 67%). The main raison for not reaching the target is that during the years less applications are received as a result of declining popularity of the action, but also due to the fact that NA being more strict in the selection procedure, selecting only the projects that are above the 48 points (success rate of 70-75%). For the call of independent eligible projects were sent in, of which 10 were selected and one reserve. In one case a project received a difference of 30 points in the evaluation by two experts, for which an additional expert was invited. According to the interviewees, the main reason for not applying is that ToI projects cost relatively a lot of energy for beneficiaries, and VET colleges in general are very busy with the implementation of own policy priorities like competence based education, leaving less time for applying for a ToI action. The ToI action also receive less applications due to the fact that tertiary level applicants could not apply for the Leonardo da Vinci programme, while this was the case in the previous Leonardo da Vinci programme. Contrary, the NA indicates that the mobility actions are not diminishing. Assessing the extent in which projects contribute to European and national priorities we refer to the figure below providing an overview is provided on how projects score on the different priorities. 210

211 Figure Number of projects funded under different priorities. Priority 6 Developing the learning environment Priority 5 Raising competence levels of groups at risk Number of projec Priority 4 Skills development of adults in the labour market Priority 3 Transparency and recognition of competences and qualifications Priority 2 Developing the quality and attractiveness of VET systems and practices Priority 1 Developing the skills and competences of VET teachers, trainers and tutors Priorities Source: CINOP (2008, 2009, 2010), Transfer of Innovation Projects Call 2007, 2008 and 2009, calculations Research voor Beleid The NA has specifically asked the project applicants to submit their application taking into account the European priorities of the specific call, on which in addition a number of current Dutch themes have been highlighted. The figure above shows that most projects in 2007 are focused on priority 2, 3 and 4, while the call for 2008 resulted in a clear focus on priority 1 and 4 projects. The call for 2009, finally show a clear focus on priority 1, 2 and 5 projects. Overall the projects cover a wide range of subjects, ranging from implementing an APL centre, developing relevant e-learning content, designing training and many more. The national agency paid a great deal of attention to whether the projects would be able to include best practice in the Dutch context, the distribution of the results and the widest possible impact. With regard the satisfaction of ToI, the interviews and reports indicate that participants are highly satisfied with the ToI action contributing to their organisational goals (see section 2.2). The impact of the ToI is firstly felt on an institutional level by broadening and strengthening the internationalisation agenda of the VETcollege and quality improvements in the college. Furthermore, beneficiaries indicate that ToI is essential in setting up and developing networks and finding partner organisations to cooperate in the future. At this moment projects are hardly able to indicate the results and impact of their ToI projects because the project selected in the 2007 call are just in the stage of finalising their project. Impact on a system level is highly ambitious to expect from this action. However, some examples are provided, such as the project called INTENT, coordinated by Kenniscentrum Handel (the Centre of Expertise for VET in the trade sector). The project results are directly addressing the needs described in the priority. With the ECVET proof certifiable unit, the project designed a VET programme with flexible devices for validation, transfer and recognition of learning outcomes. Furthermore, INTENT partners are taking part as experts in events organised at European level, connecting the project and its results firmly to the ongoing process of implementation of ECVET in Europe. Without VETPRO, internationalisation within the VET-sector would become very difficult and makes more difficult to identify innovative actions to be transferred to the Dutch context and develop international partnerships What are the main external factors affecting the outputs and results of the action? In the previous sections already a lot has been said on factors affecting the outputs and the results of the programme, such as limited visibility of ToI action, and less number of applicants leading to not reaching the targets for 2008 and 2009, the lack of cooperation between the ToI action and the DOI action, preventing a better exploitation of the programme results was also mentioned. Besides, interviewees, also indicates project related factors, such as the loss of partners during the application period or during the implementation of the project, lack of knowledge by project partners how to transfer an innovation, too ambitious targets for transfer, or a wrong balance between the time invested in the development of the innovation and the transfer. For the problem of losing a partner while implementing a project the NA introduced the rule that applicant should include 5 partners in their application instead of 3 as requested in 211

212 the criteria of the Commission. Subsequently, if one of the partners fell out while implementing the project this does not lead to a project failure. A more external factor mentioned is the economic conjuncture: the attention towards internationalisation in VET is sensitive to the economic developments. Internationalisation is not (yet) seen as core activity of most VET-colleges, therefore, when budget-cuts need to made, internationalisation is placed in the danger-zone. Some projects also face problems while implementing a project due to changing economic context. An example is the project Keep workers in road haulage industry moving, that was cancelled because of worsening economic situation with a resignation of al lot of truck drivers, target group of this project. According to the interviewees in general all the ToI projects are reaching their targets, with some exceptions Effectiveness in reaching the target groups and disseminating results How does the action contribute to the horizontal priority of multilingualism? The ToI action attributes indirectly to the horizontal priority of multilingualism since the project partners are enforced to speak foreign languages (mostly English). No project is directly related to the topic of multilingualism and the NA is not steering on this topic with their priorities and criteria. The figure below shows an overview of the country of origin of project partners. Figure Project partners by country UK TU SI SE RO PT PL NO NL IT IE HU FR FI ES GR EE DK DE CZ BG BE AT Source: Source: CINOP (2008, 2009, 2010), Transfer of Innovation Projects Call 2007, 2008 and 2009, calculations Research voor Beleid The figure above shows that partners are coming from Denmark, Belgium, Slovenia, UK, Italy, and Ireland, stating also that the number of partners from other countries is decreasing in the last year How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? The ToI action is not specifically aimed at the horizontal priority of equal opportunities (EO), other tan stimulating projects in relation to priority 5, raising competences levels of groups at risk. In total 5 projects over the years could be classified as priority 5 projects, such as electronic distance learning for disabled youngsters, risk management for European SMEs, APL for youngsters in (juvenile) prison, and C2CC in Connection to Care and career. Moreover, several projects address different target group beneficiaries. In the interviews it appeared that the NA is not steering on EO specifically by setting extra prioritises and criteria in the selection of projects. Overall interviewees consider this topic as less pressing for the 212

213 Netherlands, because the Netherlands already established a relative equal society compared with some other EU countries Effectiveness of the integration of the previous activities into the LLP? Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? What are the advantages and disadvantages of the integration of previous programmes into the LLP? Since the ToI is a new action within the Leonardo da Vinci sub programme there is no evidence of integration difficulties solved other than already described in this case study on the link between the sub programme addressing similar target groups, and the lack of cross sector fertilisation of the ToI action already described in the previous sections. Most of the remarks can be made in relation to the programme management structure. As described in relation to the relevance of the LLP in the Dutch educational sector, the responsibility for internationalisation is placed at the level of institutions. For this reason, the LLP is implemented accordingly: the Dutch government left as much freedom as possible for the institutions to implement the programme to their needs. The organisations involved in all internationalisation activities in the sub sectors and the implementation of the sub programmes in the previous programme-period, were asked to form the National Agency (Nuffic, CINOP and EP). In this way effective and efficient usage has been made of the knowledge available. In the National Agency the Europees Platform is responsible for Grundtvig and Comenius, CINOP for Leonardo da Vinci en Nuffic for Erasmus. The three organisations are responsible for the implementation, monitoring and reporting of their sub programmes. Nuffic is secretary of the National Agency. The structure of the National Agency has advantages and disadvantages for the implementation of the LLP (see table below). Advantage Disadvantage Target groups Organisatio n The organisations are traditionally rooted in the sectors the sub-programmes are targeted at. They have a good knowledge of the recent developments in the sector and know the stakeholders. The organisations are involved in other internationalisation programmes as well and are therefore capable in preventing overlap between programmes and advising the target group in the most effective way. Source: Research voor Beleid The separation of sub-programmes over the three organisations causes a lack of uniformity and coherence towards the target group. Even when considering the diversity of the target groups, the activities supported share a common objective that applies to all the target groups that can be emphasised by a more uniform organisation of the Lifelong Learning Programme. The synergy between the sub programmes is not very well developed. There have been difficulties in aligning administrative processes and communication processes across the different sub programmes. 191 Several respondents indicate that the National agency should play a more active role in interlinking the sub programmes, by linking projects that cover a similar theme or approach between the different education sub sectors. The NA could act more as intermediary role Effectiveness in reaching the target group and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? The target groups of the ToI action are all those in vocational education and training, other than at tertiary level, as well as the institutions and organisations facilitating such education and training. Overall the success of reaching target groups is average, and therefore there is room for improvement From the data available, it appears that there is a diverse group of beneficiaries representing all sort VET organisations, small and big, providing Initial VET and Continuing VET. However, most of the applications come from VET-colleges. As has been indicated before, the organisation coordinating the 191 This has been communicated by the European Commission during the Audit of April and May

214 ToI-action, CINOP, operates in the vicinity and in close contact with the VET-colleges. They exchange knowledge and experience on a regular basis, provide information folders and organise information days and workshops. Furthermore, a large share of the VET-colleges is already involved in Leonardo da Vinci projects. However, as already indicated in the previous sections the number of application is decreasing over the years that force the NA to play a more pro-active role in stimulating organisations to apply for ToI. This is a strategy that became practice since Also the thematic working groups could play a role in starting up new project ideas for ToI. The primary focus of the National Agency is on VET-colleges, with regard to other organisations, there is still room to expand the reach of target groups To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? The impact and results of the ToI-action can be identified on institutional and system level. A key role for dissemination and exploitation, besides of the project owners themselves, exists for the umbrella organisation for VET-colleges (MBO-raad) and the National Agency (CINOP). The NA place a lot emphasis on the exchange of experiences and the description for good practices. In the Jaarboek Leonardo da Vinci, best practices are presented and during workshops and in information folders, good practices are presented. However, there is still missing a clear link with the policy level for vertical D&E, involving stakeholders / decision makers that can influence national/ sectoral policies. The main issue in this respect is who is feeling responsible for further disseminating and exploiting the project results when the project stops. There are no formalised structures within and outside the LLP that have the resources, autonomy and influence for organising this process, besides the NA themselves. Often the transfer of innovation stays limited to the project level, with some exception like the project international entrepreneurship of KC Handel that changed the qualification profile on national level. Many projects are still to much focused on further developing the innovative content of the project and adapting it to their own context. Leaving less time for disseminating and further exploiting their project results (besides of transferring it into their own organisation which is one of the main goals of the ToI action). Organising the process of D&E also demands specific competences different than you need for developing the innovation, not always available in the project organisation. On project level, several beneficiaries indicate a high impact, by stating that their project results are taken on board by project partners in other countries and increasing effects in their own organisation such as more knowledge on APL in the organisation. These are examples of clear impact on organisation level. Projects indicate that the success factor for D&E is primarily related to networking activities, by being involved in all kind of partnerships, also as follow up of your project or previous partnership. Project holders also indicate that a lot of D&E is accidental. Often there is less time for D&E activities, and projects therefore value the Leonardo site, project databases, and international conferences informing other stakeholders on innovative project results. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? In general, beneficiaries are satisfied with the way the ToI-action is implemented. They emphasise that procedures could be simplified, but they are workable for most applicants experienced with applying for the ToI action. 214

215 Application In the application process, the National Agency follows the LLP guidelines. National priorities have been added. Interviewed beneficiaries indicate that sometimes these priorities have limitation for the direction applicants want to take with their proposal (interpretation differences). The advisors of the National Agency can be asked for advice by of phone. As minor flaws in the application procedure the following elements can be mentioned: - In general, it happens that the application-formats are distributed very late and close to the deadline, which causes problems for applicants, when changes are made in the format with regard to previous years. - More continuity in procedures is desirable (not too many changes in application forms on a yearly basis). - The application format of the ToI and DoI could be better harmonised. Implementation The implementation of the ToI action does not lead to problems. The National Agency is very approachable when difficulties occur. Monitoring and reporting Monitoring is carried out on a regular basis by the National Agency (CINOP). The advisors have close contacts with the project holders and keep updated by regular phone conversations or . The use of the administrative tool LLPLink has not been developed towards its full potential. Most administrators of the National Agency are having difficulties in using it and the system is at this point not very robust and stable. All organisations responsible for their sub programmes (including CINOP) have therefore duplicate book keepings, which are not integrated in one bookkeeping system for the whole LLP. The final reporting of the ToI actions is evaluated by the advisors of CINOP, and feedback is provided to the participating institutions Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? The National Agency (CINOP) has at its disposal 11 FTE to carry out the whole Leonardo da Vinci programme. The total budget of the programme is in 2009 almost 10 million. From both the National Authority (Ministry of Education, Culture and Science) and the European Commission CINOP receives half a million to implement the whole programme. CINOP is able to provide all the necessary services for implementing the programme. Judging on the experiences from beneficiaries, the coordination and implementation of the ToI action is very adequate and manageable. No shortages of any kind seem to occur in relation to the Leonardo da Vinci programme A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to the National Authority and European Commission? The data collected by the beneficiaries is analysed by the National Agency and used to supervise and support the beneficiaries. Furthermore, the data are used to serve a good practice at several occasions possible (workshops, folders, Yearly reports etc.). An interesting issue in respect to ToI projects is how to deal with assessing the impact of ToI projects, due to the fact that it often last some time before the innovation is taken up by target groups, and go beyond the project funding period. That makes it almost impossible to measure the real impact of the projects and the monitoring should therefore focus on output related aspect of the project. However, it should be possible, based on experiences of all ToI projects, to identify a set of success factors for transferring the innovation that can be used for monitoring the projects, assessing whether the project complementation comply with the conditions of good transfer. Currently such a model does not exist. This set of criteria could also be used in the selection procedure for ToI projects. 215

216 15. Grundtvig Learning Partnership : Slovenia 1. INTRODUCTION An overall purpose of this case study is to provide an in-depth insight (findings based on evidence) into key evaluation issues: relevance (including coherence and European added value), effectiveness (including contribution to the global objectives) and efficiency (including cost-effectiveness) of the evaluated Lifelong Learning Programme s action. The case study elaborates only on the Grundtvig Learning Partnership. The information presented in this case study is based on the following sources: - In-depth interviews with the national agency, the national authorities and the institutional beneficiaries of the Grundtvig Learning Partnership. In total 10 persons have been interviewed at 8 interviews (the full list of interviews is presented in the Annexes); - Desk research, which sought to analyse national legal acts, action s implementation statistics and other documentation, which was provided by the national agency, the national authorities and the institutional beneficiaries. The interviews and desk research were carried out during the period of 20 April 20 May RELEVANCE The main objective of this section is to describe the European added value of the Grundtvig Learning Partnerships action To what extent the intermediate and specific objectives of the action remain pertinent to the national policy priorities? The evidence collected for this case study indicates that the intermediate and specific objectives of the action remain pertinent to the national policy priorities. Although no clear explicit link has been made in national policy documents referring to international cooperation, the objectives of this action clearly contribute helping Slovenia developing a system of adult learning that met the strategic objectives as set on national level, by means of learning from experiences and good practices in other countries. Grundtvig Learning Partnerships are small scale cooperation projects involving adult education institutions from at least three European countries. The emphasis is on the process of establishing exchanges between the partners on specific themes and on the active participation of adult learners in the project. Particular attention is given to including people from disadvantaged groups. Exchange of practices, experience and methods contributes to better understanding of areas of common interests as well as to better awareness of varied European cultural, historical, social and economic scene. All projects are of 2 year duration. One of the partners acts as a coordinator. Cooperation with local authorities, social services, associations and enterprises is possible. Mobility of the learners is encouraged as much as possible. In order to assess whether the Grundtvig learning partnership action remains pertinent we need to assess the context of the Slovenian adult learning system and the lifelong learning strategy adopted in Adult learning was relatively underdeveloped before 1992, having an inadequate educational structure and low education attainments. In 1991 they started laying the foundations for a new system, which developed gradually and is still developing in terms of organisation and subject matter. In organising and building this new system, it was necessary to observe positive experiences, systems and achievements from past years, along with experiences from foreign countries. The outcomes after the commitment to adult education and adoption of the legislation are favourable, raising educational attainment. The patterns of participation 192 in adult education changed as well. While participation in 1987 was 27,6%, this figure reached 37% in The Lisbon benchmark, 12,5% participation of the population between 25 and 64 years of age, was reached and exceeded in (15,3% in 2005), which puts Slovenia in the 6 th place among the EU-25, after Sweden, the UK, Denmark, Finland and the Netherlands. 192 Measured as taking part in any organised learning activity twelve months prior to the interview. 193 Source: Lifelong learning (adult participation in education and training) total percentage of the adult population 25 to 64 participating in education and training, measured by the Labour Force Survey and taking into account any organised learning activity four weeks prior to the 216

217 At the beginning of 2007 a comprehensive Lifelong Learning Strategy was put forward for public discussion. Its aim is to provide all people with the opportunity for full development of their abilities. 194 The strategy's main objective is to integrate all spheres of education and training into a coherent system with an emphasis on the following core areas: (i) learning in all circumstances and periods of life; (ii) raising awareness about the benefits of learning; (iii) raising awareness about the right to learning and the shared responsibility for it; (iv) customised learning for every individual; (v) incorporating the understanding of the importance of lifelong learning in curricula; and (vi) raising levels of all types of literacy. The main object of the policy is human resources development for work and life in a knowledge-based society by modernising the system of education and training and by stimulating life-long learning. In that respect the policy is oriented on the activities related to the individuals (participants) and to the system. Specific objects related to the system development are improvement in the quality of the education system along the entire vertical and of the training system including improvement in the qualification structure of the education providers. In order to reach the above described objectives, cooperation of the local, national and European levels is therefore highly pertinent. As indicated learning partnership could play an important role observing positive experiences, systems and achievements in other countries, by sharing improvement of teaching and learning methods, best practices and innovative methods of teaching in various areas of education/learning, improving the flexibility of education/learning provision, particularly in relation to disadvantaged groups and or addressing the issue of recognition and validation of knowledge and skills obtained as a result of informal and non-formal learning. Objectives such as active participation of adult learners are also present in operational policy documents. Special attention on non-formal general education of disadvantaged groups is supported by Lifelong Learning Strategy and operationally by yearly Plan of General Adult education financed by the Ministry of Education and Sports (MES) To what extent the objectives of the action remain pertinent to the lifelong learning needs of the target groups? All kinds of institutions and organisations, whether working in formal, non-formal or informal adult education could apply for funding in the Learning Partnership Action. 195 Two main target groups benefit from the support of the Grundtvig Learning Partnerships: the adult learning institutions (as direct institutional beneficiaries) and adult learners personally (individual beneficiaries). According to the data collected for the purposes of this case study, Grundtvig Learning Partnerships remain highly pertinent to the needs of both target groups in Slovenia. The institutional beneficiaries according to the interviews indicated highly positive effects of the Grundtvig Learning Partnership action. Based on the interview with the NA, the numbers of applications for LP had been increasing each year. According to the estimation of the NA, about 25% of applicants will gain the grant in All interviewees clearly see the need of sharing knowledge and expertise in order to address topics of the previously mentioned Lifelong Learning Strategy. They all indicated that they gained new knowledge of organisational frameworks; they learned about different policies and different ways of solving similar organisational problems. As a result of previous LP they are more willing to start new, more demanding projects. interview; from 27 October 2006 based on an annual average of quarterly data instead of one unique reference quarter in Spring. oduct=strind_emploi&root=strind_emploi/emploi/em051, Adult education in Slovenia has gained an equal position in the educational system with the Resolution on Adult Education in the Republic of Slovenia by 2010 (adopted in 2004) where the need for further investments in life-long learning is acknowledged. This represents an important base for achieving higher adult education level, for raising the educational level whereby 12 years of successfully completed schooling is considered as the basis of educational standard, greater employment and further education opportunities. Therefore, Slovenia has to continue developing and modernising various educational offers that will be based on the needs of an individual and of the society, and will adapt to the needs of labour. 195 Formal learning usually takes place in schools, universities or training institutions and leads to a diploma or certificate. Non-formal learning includes free adult education within study circles, projects or discussion groups advancing at their own pace, with no examination at the end. Informal learning can be found everywhere, e.g. in families, in the workplace, in NGOs, in theatre groups, or can also refer to individual activities at home, like reading a book. 217

218 The interviews with the individual beneficiaries were not carried out for the purposes of this case study. However, several indicators allow concluding that the programme remains pertinent to the needs of the individual beneficiaries, especially for the disadvantaged groups. Several indicators, such as questionnaires answered by participants and saved by institutional beneficiaries, allow concluding that the programme is having a positive effect on adult learners language skills, intercultural competences, specialist knowledge, changes of attitudes and perceptions regarding the host country, improved perceptions regarding European integration, greater awareness of European dimension of adult education, new contacts within the field of adult education, new knowledge of different methods, new knowledge over ICT related learning methods, team skills, self-confidence and self-assurance. Though systematic data are not available, institutional beneficiaries notice improvement of interpersonal skills and more entrepreneurship of participants regarding initiatives for new international projects. Institutional beneficiaries indicated that majority of Grundtvig Learning Partnership participants are highly satisfied with their experience and are willing to repeat it. There were also questions posed to the institutional beneficiary about possibility of repeating the LP for another group of adult learners of the same institution The European added value. Would other national schemes / instruments provide enough support to learning partnerships, if there was no LLP? Would adequate attention be given to the challenges faced in meeting the lifelong learning needs? Information collected during this case study indicates that the European added value is substantial. The added value is created in several interrelated areas. In the absence of the LLP, and specifically the Grundtvig Learning Partnership action the likely level of funding provided by the national authorities to adult learners exchange would not exist or it would be exceptional. Accordingly, outputs would not be reached. The main reasons behind the inability to reach the same participants mobility results with other sources relate to: a) the lack of alternative mobility funding sources, and b) low capacity of the learners who mostly belong to the so called vulnerable groups. Grundtvig Learning Partnership programme has served as one of the main drivers for the internationalisation of the Slovenian adult education and thus contributed to the international visibility of the Slovenian adult education. This would not be achieved without the support by the Grundtvig Learning Partnerships. Internationalisation of Slovenian adult education was present also before, but mostly at the tertiary level of adult education. For lower levels of adult education international cooperation was also present, but mostly through bilateral agreements. Grundtvig Learning Partnerships is at present the main source of support for adult learners international cooperation in Slovenia. Other smaller sources existed before (National Adult Education Associations, Associations of Public and Workers Universities, Associations of International bilateral cooperation), but the capacity of these sources altogether is very low, especially after the change of the system after 1991 and reduced further in the last years of the economic crisis. International cooperation in adult education sector was possible before Grundtvig Learning Partnerships mostly for well educated groups of adults and for vocational reasons. Maybe one of the possibilities to develop similar adult learning cooperation would be to establish the adult learning programme under the national Study circle programme, which was very vivid since 1992 in Slovenia, but is nowadays vanishing due to the lack of new ideas and new beneficiary institutions who are allowed to apply for funds. However, the international and European dimension would financially not be covered through the Study Circle programme. One of the respondents among institutional beneficiaries mentioned a possibility to develop similar programmes through Europe for Citizens What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? Synergies between the Learning Partnership action, the wider Grundtvig programme and other subprogrammes could appear on different levels, such as reaching new target groups / audiences of the Programme, transferring good practices from other educational sub sectors to the adult learning sector, integration and harmonisation in the procedures, reducing duplication of efforts and administrative simplification. 218

219 There is no evidence of synergies among different sub-programmes. Institutional beneficiaries are not in advance well acquainted with other possibilities and other sub-programmes of LLP, but if they are interested they say, they find sufficient information at the National Agency. It is not that NA would not inform the applicants about all Grundtvig actions, it is just difficult to imagine for the institutional beneficiaries how a certain action, besides Learning Partnerships would fit into their project, until they start the project and elaborate on it. Through practice and experience, they gain the knowledge and ideas about possible synergies of different Grundtvig projects. Another synergy that could be better exploited concerns the target groups of the Learning Partnership action, defined as all kinds of institutions and organisations, whether working in formal, non-formal or informal adult education, overlapping potential beneficiaries of other sub-programmes (e.g VET colleges, secondary education, and higher education institutes). However, during the interviews no evidence was provided on a possible synergy or duplication. Coherence of efforts of different Grundtvig sub-actions is ensured through information conferences of NA. National authorities and responsible people of NA also meet every year before they publish the call for proposals. At the meeting they decide on priorities and the procedures. The problem is that there is very little money for other Grundtvig actions besides LP. Little money for as many as seven actions of Grundtvig decentralised actions, means that there is a lot of energy and time spent for introduction of different actions, and when only a few beneficiaries are selected, the ones who do not succeed are disappointed and are not willing to apply again in the next years in the years to come. More substantial funding for the other six actions of Grundtvig besides LP would be welcome, or reducing the number of decentralised actions to focus funding better. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities What is the progress in achieving targets against financial progress of the action? What is the extent to which the beneficiaries are satisfied with the outputs and results of the action? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The progress towards achieving the national adult learners mobility in Slovenia over the period allows indicating high effectiveness of the action s implementation: - Firstly, the funds of the action are used up to 100% each year. The number of adult learners who travelled to another country increased since 2007 according to the interview with the NA. - Secondly, according to the institutional beneficiaries responses vast majority of the participants were highly satisfied with the exchange experience and were willing to repeat it. Some made personal contacts and are still in touch through ICT. With regard the satisfaction of beneficiaries with the Learning Partnership Action, the interviews and reports indicate that they are highly satisfied with the action contributing to their personal and organisational goals. Based on the information collected during the case study, we draw the conclusion that the Grundtvig Learning Partnership strengthened the internationalisation agenda and contributed to reinforcing the contribution of lifelong learning to social cohesion, intercultural dialogue, equality concerning different generations, and personal fulfilment. It also contributed to the promotion of language learning and linguistic diversity, historical and cultural learning, an increasing participation in lifelong learning by people of all ages, including those with special needs and from disadvantaged groups. Furthermore, beneficiaries indicate that learning partnerships is essential in setting up and developing networks and finding partner organisations to cooperate in the future. At this moment learning partnerships are hardly able to indicate the results and impact of their projects because the project selected in the 2007 call are just in the stage of finalising their project. Impact on a system level is highly ambitious to expect from this action. Slovenian National Agency expresses confidence that such data will soon be available, which could reflect the importance of all these achievements to the beneficiaries. This year a new action specific 219

220 questionnaire was prepared and all beneficiaries were requested to fill it in. Questionnaire includes approximately 30 questions for each action and seeks to equally address both broad and narrow issues of the programme. Meanwhile, the match between beneficiaries needs and their satisfaction with project outputs and results is ensured through the regular contacts between beneficiaries and the National Agency (such form of communication is quite efficient as there are approximately only 10 LP projects each year in Slovenia) and various events organised by the National Agency. Current data do not provide evidence of mismatch What are the main external factors affecting the outputs and results of the action? Desk research and interviews indicate that the outputs and results of the Grundtvig Learning Partnership are mainly affected by internal factors associated with programme management, such as the financial provision, good management and clarity of requirements for applications, transparency of selection criteria, award process, clarity of expenditure eligibility requirements, timeliness and clarity of project reporting requirements. Other than that, outputs of the action during the 2007/2009 period were negatively affected by factors of a personal matter. Some individual beneficiaries could not travel due to being in position to take care of ill family members, or they did not feel comfortable with being placed in different hotels, while visiting another country. One of the institutional beneficiaries proposed that after being selected for the project they wished that there would be more countries and institutions involved, which were denied participation from their NA, while they felt they could benefit to the project if they were involved. Neither beneficiaries nor NA were able to identify external factors which affect the outputs Effectiveness in implementing horizontal policies How does the action contribute to the horizontal priority of multilingualism? We can conclude that the Grundtvig Learning Partnership contributes directly to the multilingualism objective by improving the language skills of the participants. The horizontal priority of multilingualism is integrated into the calls for proposals. - On the one hand, the priority is given to the participants that have some knowledge of English or/and other languages of partner institutions. - On the other hand, additional language courses for individual beneficiaries are a possible incentive and an option subsidised by Transversal KA2 action. The information collected during the case study analysis suggests that integrated approach could be more effective at the national level. The institutional beneficiaries are not familiar enough to the differentiated approach (have insufficient knowledge on the Transversal KA2 action). As well, differentiated approach is not so much in use, since there are limited resources provided for this purpose. Study visits and participation in Grundtvig Learning Partnerships is an important incentive to adult learners to improve their language skills, to test their language skills and to decide to use it in everyday communication as well as to improve written language through reading literature and through ICT technology. Adult learners are usually learning language, while focusing on project preparation, and intend to enrol in a language course later, after the project is closed. On the one hand, adult learners without linguistic skills generally cannot take part in the projects. Participating in the project generally ends up in a beneficiary s increased interest for other European cultures and for learning languages. In case of the socially marginalised target groups there are many follow up contacts among the participants increasing their language skills. 220

221 How is the implementation of equal opportunities (EO) ensured by the action evaluated? Are EO issues integrated in the accepted project applications? Are EO issues addressed during project implementation? Overall, the issues of equal opportunities are implemented on average. More systematic approach should be taken towards implementing the equal rights of adult learners. Adult learners, which are not involved in any kind of non-formal education, are very difficult to reach. There are voluntary organisations which deal with the problems of disadvantaged groups. Research on educational needs of these disadvantaged groups would be welcome to find ways of reaching them. Integrated approach is implemented in the case of ensuring gender equality by delegating the responsibility to ensure gender equality to the institutional beneficiaries. Equal rights of men and women are not prioritised in the Grundtvig Learning Partnership because of the common practice to treat women and men equally. Generally it is difficult to involve equal number of men and women since women are much more active in adult education in Slovenia, as educators or participants. Based on the interview with the institutional beneficiary there are some problems in case of disabled persons participating in the Learning Partnership action. In case of mobility of persons with disabilities, the number of visiting participants per project is decreased. This means less mobility and is not a very good solution according to the reports of institutional beneficiaries. Students with disabilities should be able to apply for an additional scholarship covering the necessary medical expenses or the costs of the following person. An insufficiently addressed problem relates to the mobility of the participants from socially disadvantaged background. As it was noted earlier, these participants have fewer possibilities to take advantage of the Grundtvig Learning Partnership. In Slovenia where living standards are still relatively lower than in many EU15 Member States, this is a significant problem. No priority is applied at the national level that would provide support for the socially disadvantaged participants. They are usually formally less educated and do not consider to engage in continuing education, unless it is for some practical or personal reasons. In projects addressing the socially marginalised target groups the issue of equal opportunities is being well taken care of. To illustrate this point, in projects involving older people empowerment targets are present at different levels (on the level of content, methods, didactic approach etc). The issues are integrated in project application and in the project implementation documents. They are also integrated in the dissemination activities Effectiveness of the integration of the previous activities into the LLP. Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? Overall, judging from the data collected under this case study analysis, the advantages of the integration of the previous activities into the Lifelong Learning Programme outweigh the disadvantages created. There is sufficient evidence that the integration of the LLP was successfully implemented at the national level. Actually it was already integrated before the official integration at the European level, because of specific situation (small state) where it would be too expensive not to integrate it. The major content-related benefit (related to Grundtvig Learning Partnership) of the integration of the previous activities into the LLP is undoubtedly the possibility to share project outcomes of different subprogrammes or within the Grundtvig programme in order to overcome content related overlap in the programme. The other advantages of the LLP integration relate to the management-related benefits for the decentralised actions on both national and institutional levels: - Firstly, the integration has allowed reaching wider target groups with lower cost and has reduced the duplication of efforts in disseminating the information about the separate sub-programmes of LLP. The integration of the LLP at the national level has been fully implemented: the National Agency operates as a single one stop shop for all sub-programmes and this has allowed for better coordination, dissemination of results and other information and created cost-efficiency in the administration of the sub-programmes. 221

222 - Secondly, the integration of the LLP, according to the interviewees, has encouraged closer and more direct involvement of the national authorities into the programming and implementation of the programme. However, all these integration effects were of very limited scale and extent as the previous programmes (Socrates and Leonardo da Vinci programme) were already managed by the same agency. As a result, participants of various events and seminars organised by NA in a first row were introduced to the LLP as a whole and only then encouraged to chose actions which suit their needs best or participate in a few actions Effectiveness in reaching the target groups and disseminating results How successful is the action in reaching the target groups and institutional beneficiaries? According to the data collected for the case study analysis, the Grundtvig Learning Partnership action is rather successful in reaching both the institutional beneficiaries and the individual beneficiaries since the numbers of submitted applications significantly outweigh the amount of grants (see the table bellow). Table Number of applications and grants Applications Grants Applications Grants Applications Grants Number Amount in EUR Source: data from Slovenian National Agency. On the other hand, the dissemination of information about the programme to the beneficiaries should be improved. Data available from the Slovenian National Agency does not display constant growth in numbers of submitted applications. Furthermore, some Slovenian regions are still geographically and organisationally underrepresented in terms of application submissions. In 2008 Slovenian National agency did not receive a single application from beneficiaries located in Koroška and Zasavka regions To what extent are the Programme results adequately disseminated and exploited (at both horizontal and vertical levels)? The main finding is that the dissemination and exploitation of Learning Partnership results are moderately sufficient well organised, with the notion that NA could put more emphasise on analysing the content of learning partnerships in order to draw valuable lessons for policy making and play a more active role in disseminating and further exploiting these lessons. In case of the Grundtvig Learning Partnership the results of the action can be defined as: a) The positive experience of participating adult learners in projects of Grundtvig Learning Partnership, their new knowledge of language and increase in intercultural competence, b) The effects participation in the programme had on the internationalisation and the quality of andragogical approach as well as on its management strategy of adult learning organisations. At the national level, the National Agency disseminates, presents and publishes monitoring data and examples of best practices. Moreover, they organise meetings and discussions with the National Authorities, Adult Education Institute, and other organisations for ensuring a wider dissemination of information on the possibilities offered by the programme. Despite detailed information collected about Grundtvig Learning Partnership from the institutional reports, the NA does not perform any qualitative analysis on the results of the sub-programme and its impact on the national system of education and training. No information on the sustainability or exploitation of project results is gathered after the contracts finish. The institutional beneficiaries themselves are responsible for the dissemination of the results. There are no common national guidelines on how the results should be disseminated; hence each institution has come up with their own instruments. The most common solution is the informal dissemination of reports about visits in adult education newsletters or magazines, internet websites, local radio and TV. Local newspapers are also 222

223 willing to publish articles on study visits of participants from other countries. Another common measure is organisation of the informational events, such as Adults learners week, and different conferences where some Grundtvig Learning Partnership project coordinators and sometimes participants present their experience. One of the Grundtvig Learning Partnership projects is every year awarded a national award by the minister of Education for the best quality of the project under the title Apples of quality. 4. EFFICIENCY The main objective of this chapter is to determine the gaps in the functioning of the implementation, monitoring and control system of the LLP that should be improved in order to increase the cost-effectiveness of interventions. Overall, available data indicate moderate efficiency of the implemented action as there is a lack of resources to ensure sound management of the programme and the inadequacy in use of the monitoring data to improve it Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? In general the efforts to harmonise and simplify the Grundtvig Learning Partnership management and control system are welcomed by the national stakeholders and there is an overall agreement that the situation is satisfactory. The major complaints are targeted to relatively unstable management system and the constantly changing rules and administrative documents relate to the period. Institutional beneficiaries report no complains on the management and control system Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? In the Yearly NA Reports there is a clear trend to complain about constant work overload. The information collected during the case study analysis suggests that the administrative workload is caused by lack of competence and time at the NA to evaluate results of Grundtvig Learning Partnership. On the other hand, the Slovenian NA has only one person to deal with Grundtvig sub-programme and all of its actions. Therefore complaints about administrative workload are rather reasonable. Interviews with a representative of Slovenian NA have revealed that the main issue is the time wasted on administration of actions which receive only 1-2 grants. It is especially the case with Grundtvig Senior Volunteering Projects and Grundtvig Workshops which target problematic target groups. Dealing with such projects is not only time-consuming but also requires shift of attention from project implementation to project administration. To deal with the work overload, the National Agency is planning to involve external evaluators for a survey of level of satisfaction of individual beneficiaries after the projects are finished A lot of Monitoring information is being generated by beneficiaries (through their monitoring reports). How is it used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? Judging from the data collected for the case study analysis, most of the data collected through the monitoring reports of the institutional beneficiaries is rarely used for other purposes except reporting to the EC. This illustrates the point earlier made that there is a world to win better exploiting the results of Learning Partnerships. 223

224 16. Grundtvig In-service Staff Training for Adult Education Staff : Denmark 1. INTRODUCTION This case study investigates the implementation in Denmark of a specific action under the Grundtvig programme, the scheme of individual grants to adult education staff for participating in in-service training courses abroad. This is one of the Grundtvig actions being managed at the national level, by the relevant public authority in each participating country. In Denmark the action is managed by the Danish Agency for International Education, which is part of the Ministry of Education. Teachers, guidance officers, managers and others engaged in professional practise within adult education may apply individually for these grants. The courses should last a minimum of 5 whole days and a maximum of 6 weeks. There are three deadlines for application each year, and the maximum grant for each person is The grant may cover travel expenses, subsistence and course expenses. Participants are selected on the basis of the European and educational quality of the proposed course and from an assessment of the chances that knowledge and benefits from the course will be disseminated in the environment of the participants. In order to secure distribution on a national basis only one person is normally granted participation in each course. After the course each participant has to hand in an evaluation report to the national agency. Because of the specific nature of this Grundtvig action the emphasis in this case study is on the individual beneficiaries. Four beneficiaries of individual grants, representing three different sectors of Danish adult education, were interviewed. Afterwards evidence from the organisational environments of three of the beneficiaries was acquired through interviews with other members of the organisations. Furthermore an official from the relevant national authority (the Ministry of Education) and a national level social partner were interviewed. The case study was carried out in the period of 1 May and 21 June RELEVANCE The main objective of this section is to describe the European added value of the selected sub-programme / action To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the national policy priorities? What are the links between the LLP and national policies? Policy documents for lifelong learning as well as for internationalisation of education confirm that the Grundtvig programme objectives, as well as the specific scheme for in-service training of adult educators, are pertinent to the Danish policy objectives. This is confirmed by the official from the Danish Agency for International Education as well as by the Chairman of the Association of popular adult education, both of whom were interviewed for this study. Improving the education of educational staff, particularly through their continuing professional development, has been recognised in the Education and Training 2010 Work Programme. The objective 1.1 of the 'Education & Training 2010' work programme - Improving education and training for teachers and trainers- highlights the importance of providing conditions to support teachers through initial and IST activities (Council of European Union 2007). The Commission s study on adult learning professions also emphasises the importance of IST activities. 196 The general objective of the Grundtvig sub-programme is to respond to the educational challenge of an ageing population in Europe and to help provide adults with pathways to improving their knowledge and competences. This general objective is further assisted by some specific objectives, of which the following are in particular relevant for this case study, namely the development of innovative adult education and management practices and to encourage widespread implementation, and secondly the support of innovative ICT-based educational content, services and practices. 196 Research voor Beleid (2008) ALPINE Adult Learning Professions in Europe, A study of the current situation, trends and issues. 224

225 National adult learning policies in Denmark reflect these objectives as well. The Danish Government has formulated four main objectives for the formal adult education: (1) to motivate more people to participate and strengthen the recognition of prior learning, (2) to strengthen the general basic skills for those with the lowest level of education and other vulnerable groups, (3) to make adult education attractive, targeted and flexible with reference to employers and employees and (4) to financially support the initiatives undertaken. 197 In order to reach these objectives it is important to invest in the quality of provision in the adult learning sector, the quality of the staff involved. It is the adult learning staff, who facilitate the learners in the development of their knowledge, competences and skills, whereby their quality are of vital importance. Quality in this respect refers to knowledge, qualifications, competences, skills and motivation of all staff involved in adult learning. Not only teachers and trainers need to have such qualities, but all persons involved in delivering or opening up learning opportunities, including management and administration, guidance personnel and mentors. Here, one see a clear link with the objective of the Grundtvig IST action, which is to help improve the quality of lifelong learning by enabling persons working in the field of adult learning, in the broadest sense, or who are engaged in the IST of such persons, to undertake a training course in a country other than that in which they normally live or work, addressing the content and delivery of adult education and teaching methodology; accessibility of learning opportunities for adults, in particular for disadvantaged social groups; management of adult learning, including governance at local and regional level, administration, quality assurance, support services such as counselling and guidance, developing community-based schemes for adult learning etc; and the system/policy-related aspects of adult education including all types of strategic issues, funding models, development of cooperation between providers in the context of learning regions, indicators and benchmarking etc. For the last three decades internationalisation of the educational system has been an explicit objective in Danish education policy. Both administrative and strategic units have been organised to pursue this objective. The most recent report to the Danish parliament was published in It stated a number of goals for the government s internationalisation strategy, among them to ensure that education gives Danish students and employees skills to do well in international environments and to support Danes in studying, researching and working abroad (Ministry of Education 2004). Consolidating the network of public and nongovernmental institutions working with adult education is crucial in ensuring access to mobility of educational staff pursuing in-service training To what extent the intermediate and specific objectives of the sub-programme / action remain pertinent to the lifelong learning needs of the target groups of the programme? The general picture from the Danish data is that the action is relevant to the learning needs of adult educators. The need for In Service Training activities is becoming more relevant, especially considering the changing context in which adult learning staff is working. Studies show that all over Europe important changes occur that influences the NVAL profession (e.g. audience, content, and methods). First, the learner populations are changing through demographic developments (such as increasing migration, greater life expectancy, higher levels of education and a willingness to assume more of the costs by participants in their own learning process). Secondly, studies show that issues like environment and health have become more important, just like the areas of management, economics and new media. The range on offer has also become more greatly differentiated in the language area. New fields of activity, such as educational counselling, supervision and coaching may be added. Besides, educators of adults have to create learning situations that fit with the learners who are as group increasingly heterogeneous and self-directed. More learner-oriented methods, new forms of methodological change, and the combination of different learning locations and learning methods are more and more important. Moreover, e-learning is coming to play a significant role among the methodological tools used in the NVAL field. The same is true for other types of professional activities. Management staff, for example, has to be aware of the great variety of organisational forms and of the different approaches that can be applied. They have to deal with changing contexts like the decline / increase in government spending on adult learning and they have to reflect the aims of their organisation in the light of the needs of their target groups / learners. Complexity and diversity also characterise the task profile of administrative staff who have to cooperate with trainers, programme planners and with the management, and who act at the interface between the institutions and its clients. 197 Eurydice homepage: 225

226 The situation in Denmark is not different. An evaluation report (2005) from the Institute of Evaluation in Denmark points out that there is a need for improved competence among the teachers at the Adult education centres to meet the weak participants, the participants from homes without any study experience or participants with another home language than Danish. This goes also for the study and career counsellors at the Centres. The various needs of these groups are to be identified in order to ensure equal access of all these groups to mobility in in-service training. Some of the needs of education providers, identified in a previous study, are: - In-service training to be able to handle participants with a low motivation, including early school leavers. This need is especially pressing for elderly adult education staff working with younger learners. - Increasing professional competences in the liberal adult education system. - Skills of handling educational needs of learners with special needs (currently this responsibility is transferred from municipalities to adult education centres). - Introducing flexible and ICT-based learning. - Increasing linguistic and ICT competences of educational staff. - Resolving problems resulting from a shortage of teachers in certain fields, including mathematics and natural sciences. 198 The general picture from the Danish data is that the action is relevant to the learning needs of adult educators. One source for this is the written evaluation reports produced and handed in by course participants after completion of the courses. According to the official from the Danish Agency for International Education the reports document that participants are generally very satisfied with the programme. There are no statistics available to confirm or disprove this, but the examples of evaluation reports we had to opportunity to look at in the agency office seemed to confirm it. The four adult educators we interviewed had participated in the following types of courses: - A five-day conference on e-learning and mobile learning held in Cyprus; - A five-day seminar on methods of lifelong learning held in Cyprus; - A four-week course on Italian language and literature held in Italy; - A five-day course on prevention of drop-out and school disaffection held in Spain. Three of the adult educators stated that that the courses abroad which they had attended through Grundtvig grants were relevant to their needs. The fourth participant stated that the course theme was relevant, but that the usefulness was limited by lack of quality. Overall, the courses reflect the need to promote ICT-based learning and linguistic competences. The need for skills of group management, flexibility in working with learners with special needs, as well as skills in specific subjects could potentially be met in mobility schemes as well What is the European added value? Would other national schemes / instruments provide enough support to the activities funded by the evaluated LLP sub-programme / action, if there was no LLP? Would adequate attention be given to the challenges faced by the disadvantaged citizens in meeting the lifelong learning needs? Information collected indicates that the European added value is high only in some fields of adult education. More precisely, LLP activities allow educators to increase their linguistic and ICT skills and acquire a European perspective on their work. Although, Denmark have a comparatively well developed system of teacher training structural measures to promote lifelong learning among adult learning staff are missing. Training of adult learning staff in Denmark takes mainly place on organisational level. Organisations have their own in-service programme which means that they choose to train and develop their own adult educators, although not being required to do so. However, training can also take place outside an organisation, universities or other trainer institutes. Examples of training activities provided by other institutes are: - Teacher training colleges, providing general teacher training (4 years), 198 Ibid, p. 3, 6, 8,

227 - Centres for further education (CVU) and the Teacher s training colleges, - The Danish University of Education (DPU). The four adult educators interviewed as part of this case study all stated that if support from the Grundtvig programme had not been available they would not have been able to participate in such teacher training courses abroad. Regarding alternatives, one mentioned that funds from a local foundation had earlier been received, another (who had participated in an extended course in Italy) said that the Italian state sometimes offered free courses to foreign teachers, and a third said that some funds for in-service training were available within the organisation itself; but in all three cases these alternative sources would have been insufficient to participate in the kind of courses available through Grundtvig. In addition to the LLP, Denmark participates in the Nordplus programme, organised by the Nordic Council of Ministers and covering the Nordic and the Baltic countries. The Nordplus programme aims to promote Nordic languages and culture and mutual Nordic-Baltic linguistic and cultural understanding as well as to strengthen and develop Nordic educational cooperation. The programme includes an adult learning action focused on developing cooperation and networking between contributors to adult learning (education institutions as well as enterprises, organisations and NGOs) in order to promote the development of quality and innovation in all adult learning (quoted from Nordplus home page). Nordplus does not include a scheme of individual grants for in-service training abroad. At a general level it pursues many of the same objectives as the Grundtvig programme, but only for the Nordic and the Baltic countries. The chairman of the association of popular adult education stated that the Grundvig programme had been instrumental in widening the perspective of many adult educators from the Nordic to the European, but that there is still much potential for further European collaboration in this area. Danish education policy generally gives high priority to the lifelong learning needs of disadvantaged citizens. There is a range of free or cheap educational opportunities available for citizens with limited economic and educational resources, as well as for citizens with different kinds of disability. Education for these groups is provided both within the public and the liberal adult education systems. 199 European exchange projects may facilitate the circulation of good practice examples and catalyse development of methods for flexible and ICTbased teaching. If the Grundtvig individual grants for in-service training abroad did not exist some national support for participation in such courses could probably be obtained, given that Denmark has a strategy for internationalisation of education. However, the funds would probably be smaller and selective. Therefore synergies could be identified and duplication could be avoided by using LLP initiatives only in the areas not covered by national and Nordic schemes What evidence exists of synergies between different sub-programmes? Do duplications, overlaps or other disadvantageous issues exist between the sub-programmes? Were there preconditions for coherence of efforts? Based on the information obtained for this case study, synergies among the different sub-programmes are largely unexplored, and awareness of both synergies and duplications is limited. The interviews with course participants provide one example of how synergies between different sub-programmes can be developed. Inspired by the Grundtvig activities of the director of one school, another teacher has acquired support from Nordplus for a preparatory visit to Greenland, which resulted in a partnership agreement. In the interviews with individual beneficiaries we asked about synergies or duplications in relation to other parts of the lifelong learning programme, such as Leonardo or Erasmus. However, none of the four persons had comments to offer on this. They seemed to have little knowledge or awareness of these other programme elements. The availability of a number of schemes that resemble each other may also make decisions difficult for individuals and organisations. The official interviewed in the Danish Agency for International Education found that the increased number of services might be difficult for organizations and citizen to figure out which services fit their particular needs. The social partner interviewed (representing the national association of popular adult education), who was also involved in policymaking at EU level, perceived the synergy effects within the lifelong learning programme as surprisingly limited. He saw few really transversal initiatives. But 199 Andersson, p

228 he also noted that the structures and cultures of education held many barriers for this; for instance universities and institutions for adult education often have little in common. 3. EFFECTIVENESS The main objective of this chapter is to provide proposals on how to increase the extent to which the (vertical and horizontal) objectives and targets of the action could be achieved Effectiveness in achieving targets and contributing to priorities: What is the progress in achieving action / sub-programme targets against financial progress of the programme? What is the extent to which the beneficiaries are satisfied with the outputs and results of different sub-programmes and actions? What explains the match between the needs of the beneficiaries and their satisfaction with the outputs and results? The objective of the Grundtvig programme is to improve the quality of lifelong learning by enabling persons working in the field of adult learning, in the broadest sense, or who are engaged in the in-service training of such persons, to undertake a training course in a country other than that in which they normally live or work. Thus, the target group for Grundtvig programmes is these persons engaged in the provision of adult learning whether they work in an adult education organisation or a public or non-governmental organisation. According to the records of the Danish Agency for International Education the number of applications and the number of grants given to individual adult educators during the years was as follows. Table Number of applications and grants Applications Grants Applications Grants Applications Grants Number Amount in EUR Source: National Agency records The largest number of application was received in The number received in 2008 and 2009 was somewhat lower, but the number of funded applications rose. This can reflect both an improvement in the quality of applications and an increased priority given to the action. As mentioned earlier the National Agency would like to see a higher number of applications. All in all 71 grants were given. Table Share of grants, in EUR Grants for in-service training courses Grants for all nationally managed actions Source: National Agency records The share given to participation in in-service training courses rose from 6% in 2007 to 10% in So the course grants seem to have been given somewhat higher priority, especially in According to the official in the national agency tells me that all funds allocated to the action are given in course grants each year. However, a small proportion remains unused because of those beneficiaries who late in the year (when reallocation is not technically possible) do not use their grants, for instance because of illness. In 2009 the allocation was and the amount actually used was , which is 98%. An increasing number of individual adult educators have thus participated in LLP from Though this period of time is short it indicates that the distribution has been improved by reaching an increased number of individual adult educators. From the records of the agency it is possible to see in which European countries the course for each person took place in (note that this is based on the actual list of participants, where the total number adds up to 73). The four top receiving countries are United Kingdom (12 persons), Italy (12 persons), Greece 228

229 (9 persons) and Cyprus (9 persons). If the receiving countries are grouped according to the European regions (as defined by the UN) the distribution is as follows: - North Europe: 20 - South Europe: 44 - East Europe: 8 - West Europe: 5 It should be noted that in the UN definition of regions the Baltic countries (receiving 3 persons) are part of Northern Europe. The distribution shows that Danish adult educators primary participate in education and training provided in south European countries. There may be different reasons for this. For instance lower cost levels in the South may make course participation comparatively cheap for educators from the North. The available information does not allow any conclusions on this. Another dimension is the type of institutions where the grant receiving adult educators worked. Adult education in Denmark is organised in three main sectors: General adult education, where adults can attend teaching in general school subjects at lower and upper secondary level; vocational education and training for adults; and popular adult education which includes evening classes in subjects of general interest as well as folk high schools. On the basis of the records the distribution of grant receivers appears as follows: - Centres for general adult education: 14 - Adult education associations: 14 - Vocational education and training institutions: 8 - Language schools for immigrants: 8 - Folk high schools: 5 - Schools for adults with special needs: 5 - Other institutions: 19 The category other institutions includes a mixed lot of museums, documentation centres, private course providing organizations and others. An internet search reveals that some of the institutions no longer exist and seem to have been small-scale initiatives. The list shows that the Grundtvig grants are used by all types of adult education institutions and that no single sector dominates. This variety confirms that the target group has been defined in the broadest sense as prescribed by the Commission; both public, semi-public and private adult education providers have been awarded. According to the official in the Agency for International Education the direct beneficiaries the adult educators participating in courses abroad with Grundtvig support - are very satisfied with the program provided. This perception is verified by the written evaluation reports from each participant. Benefits indicated in these reports include increased interest for own teaching subjects, improved language skills, increased awareness and reflection on own teaching practice, improved professionalism, new ideas for teaching materials and didactic methods, knowledge on other countries, other cultures and educational systems in other parts of Europe, motivation for professional development and lifelong learning, increased career opportunities, knowledge on possibilities for financial support from the European Union. Of the four course participants interviewed in this study, three confirmed that they were very satisfied with the courses and that participation benefited their professional work both as teachers and in other roles (management, development). The fourth participant was less satisfied and stated that the course did not live up to expectations. The quality of contributions was insufficient and the organisers seemed more interested in promoting local partners than in establishing a really cross-national space What are the main external factors affecting the outputs and results of the sub-programme / action? One important factor is knowledge. The official in the National Agency held the view that many institutional beneficiaries at local level are without knowledge of the possibilities in the Grundtvig programme and that this has resulted in fewer applications for grants than wished for by the national agency. The four course participants interviewed for this study obviously had knowledge of the programme; but one of them said that she would not have had knowledge of the course unless a colleague (who had been given special responsibility for circulating information in international opportunities) had pointed it out to her. Both the 229

230 National Agency and the national associations of adult educators disseminate a considerable amount of information on the programme, but it seems to be difficult to get the attention of the adult education staff. A main cause of this is probably the internal structure and culture of educational institutions, where teachers are often strongly focused on immediate teaching tasks and less open to general or transversal information, even if this information may in the end be relevant to them. The chairman of the association of popular adult education says that the will of adult education institutions to involve their staff in international activities is an important precondition for best use of the Grundtvig actions. In popular adult education this has been difficult because both activities and organisations have often had an informal character. Another factor is the general economic situation. The national agency thought that the current economic difficulties, which impact on both the private and the public sector, has consequences for many of the potential institutional beneficiaries in Denmark. In spite of the covering of direct participation costs through Grundtvig it is increasingly difficult for adult education institutions to cover costs related to the absence of teachers being abroad on training courses (e.g. costs of substitute teachers and to cover teachers working hours while they are abroad). The national agency has experienced that institutional beneficiaries often find it expensive to have teachers going abroad due to the number of working hours typically tied up in training course participation, while the single adult educator does not want to spent personal holidays on training and courses. In the interviews with the course participants, however, these difficulties were not very evident. Only one of them, the teacher participating in a four-week language course, mentioned difficulties in getting acceptance and supplementary funding in his organisation as a barrier for participation. One could expect language to be a barrier for participation, and the language skills do vary between participants. However, according to the official in the Danish agency, Danish participants are generally good at communicating in English (normally used as lingua franca) compared with participants from other Member States. In fact two of the adult educators interviewed mentioned limited English skills among course participants from other European countries (for instance Spain) as a difficulty in the courses. In sum the internal structure and culture of educational institutions appear as the most important external factors Effectiveness in implementing horizontal policies: How does the action contribute to the horizontal priority of multilingualism? The action contributes to the advancement of multilingualism, but in specific ways. It is necessary here to distinguish between language courses and courses on other topics. Regarding language courses, the official in the national agency pointed out that the agency does not give grants to courses in foreign language skills as such; courses must have a focus on methods and/or materials for teaching the foreign language. Because English skills are widespread among Danes, the agency is more inclined to give grants to courses in teaching other foreign languages than English, for instance courses in the teaching of Spanish or Italian. One of the informants in this study had participated in a language and literature teaching course in Italy and stated that this kind of course was necessary for him to maintain his Italian teaching in general adult education. The chairman of the association for popular adult education also mentioned that Grundtvig support had often been used by foreign language teachers. In this way the action contributes to the continued presence of different European languages in the Danish adult education system. The other three informants had participated in courses on other topics with English as the work language. The participants are mostly experienced professional adult educators with fairly good English skills. One benefit for them is learning to communicate cross-nationally about specific professional issues. It is a condition for Grundtvig support that the courses attended include participants from at least three nationalities. A participant who teaches and develops e-learning says that through participation in an e- learning conference he got used to handle the relevant concepts in English and in communication with professionals from other European countries. Another way is getting used to communicate in a lingua franca usually English with people from different language groups to whom this language is also their second or third language. One participant says that it can be problematic when everyone speaks a foreign language, but that you have to be patient. In sum the action contributes to multilingualism for some participants, to intercultural communication skills for others. 230

231 Is the implementation of equal opportunities (EO) satisfactory in the programme? The individual beneficiaries had very few comments on this. One said that in the course she attended there was a satisfactory balance of gender and also of nationalities represented. An aspect of equal gender opportunities is the gender distribution of the individual recipients of Grundtvig grants for in-service training courses. For the years the distribution of Danish individual beneficiaries was: - Men: 28 - Women: 45 Teaching in most types of adult education is generally done more by women than by men, so there is no reason to assume that the distribution differs from the recruitment base for the courses. The official from the national agency commented that the agency does not regard an equal distribution according to gender, ethnicity etc as a target in itself. The agency tries to give support where it is most needed, not by individuals but by institutions and organisations of adult learning. Thus the agency s approach to equal opportunities reflects the strategy of focusing on dissemination of benefits from course participants to their organisational environments. One aspect of individual equal opportunity that the agency upholds is special support for course participants with some kind of disability. The chairman of the association of popular adult education had the impression that equal opportunity related to ethnicity and immigration had been an important issue in several Grundtvig projects, and that the projects had supported innovation in language teaching for immigrants Effectiveness in integrating previous activities into the LLP: Is there any evidence of integration of education and training across sectors and sub-programmes, or any evidence of integration difficulties solved? Has the integration of the previous activities influenced the target groups? What are the advantages and disadvantages of the integration of previous programmes into the LLP? This study is focused on a specific action within the Grundtvig programme. The beneficiaries interviewed individual course participants and representatives of their organisations have given no comments on consequences of integration previous activities into the Lifelong Learning Programme. Some of them reflect on the relationship between different Grundtvig actions, and saw participation in courses as a way of developing contacts that may lead to the establishment of partnerships, projects and networks. None of them reflected on relationships to actions in the other programmes (like for instance Leonardo da Vinci). The official from the Danish NA commented on the development of the different parts of the Grundtvig programme. Traditionally there had been main services: support to in-service training, learning partnerships, and preparatory visits. Today the number of services has been extended to seven with the addition of senior volunteering projects, workshops, assistantships, and visits/exchange. She found that the increased number of services might be difficult for organisations and citizen to figure out; what specific service does fit their particular needs? The chairman of the association had the impression that integration across sectors and sub-sectors in the LLP was up to now limited. He thought that the way forward was not to create new programmes but to strengthen and broaden existing programmes Effectiveness in reaching the target groups and disseminating results: How successful is the LLP in reaching the target groups and institutional beneficiaries of subprogrammes? According to the data collected for the case study analysis, the success of the action in reaching the target groups has been average. The target group of the action investigated here consists of individual beneficiaries adult education staff and the organisations they work in. At one level the action seems to be successful in reaching the target 231

232 group of individuals; as documented above (section 3.1.) staff from a broad range of Danish adult education organisations have used grants from the action during the last three years, and the persons interviewed all say that the courses have been relevant to their needs. At another level it seems that more people in the target group could have been reached. Both the official in the national agency and the chairman of the association of popular adult education say that more people in the target group could have been aware of the scheme and could have applied for grants. The reasons for this have been discussed above (section 2.1.2); the main reason seems to be culture and structure of adult education institutions To what extent are the Programme results adequately disseminated and exploited? The training for which grants are awarded must relate to the candidate s professional activities in any aspect of adult learning, which may have to do with: - The content and delivery of adult education, in particular course content and teaching methodology; - The accessibility of learning opportunities for adults, in particular for disadvantaged social groups; - The management of adult learning, including governance at local and regional level, administration, quality assurance, support services such as counselling and guidance, developing community-based schemes for adult learning etc; - The system/policy-related aspects of adult education including all types of strategic issues, funding models, development of cooperation between providers in the context of learning regions, indicators and benchmarking etc. To reach the goals of the Grundtvig programme a dissemination of results between institutional beneficiaries but also from the individual adult teacher to home organisation is fundamental. This concerns especially activities aimed at management of adult learning, including governance at local and regional level, administration, quality assurance, support services such as counselling and guidance, and developing community-based schemes for adult learning. The same goes for activities aimed at improving the system/policy-related aspects of adult education, including all types of strategic issues, funding models, development of cooperation between providers in the context of learning regions, and indicators and benchmarking. The Danish NA especially emphasises the institutional context of appliers for grants. In this respect the Danish implementation of LLP differs from implementation in other Member States (at least according to the Danish NA). The philosophy behind the Danish strategy is that as funding and resources in the Grundtvig programme are limited it is important that those adult educators receiving grants share their results from programme participation with their home organisation. The individual participant should become some sort of an ambassador to LLP in own organisation in particular and in society in general. The agency has put effort into information strategies aimed at the managerial level of organisations to improve managers knowledge on and attention to the Grundtvig Programme. The chairman of the association says that the agency is doing a good job of dissemination information. Other actors also do this; for instance the association of popular adult education has a separate section on international activities and opportunities at its home page, and its weekly newsletter often includes international issues. According to the official at the Danish Agency for International Education the Grundtvig programme gives individual participants opportunity to reflect on own teaching practice and working condition, reflections which are motivated by stories from other participants. The agency assesses that results from participating in the Grundtvig sub-programme are significant at individual level, but that the institutional learning processes inside institutional beneficiaries are limited. Difficulties fulfilling the goal of institutional learning generated from the Grundtvig programme are as the agency perceives the situation due to the weak attention on the Grundtvig programme profits paid by directors and educational managers of institutional beneficiaries. The interviews with course participants seem to confirm this. Three of the four persons say that they have disseminated knowledge and inspiration from the courses in their organisations (the fourth person found the course unsatisfactory). New knowledge has been circulated and other educators in the organisations have been inspired to take up international activities. However, the initiatives have generally come from the persons themselves, not from their managers or other parts of their organisations. The managers have not taken initiatives to secure dissemination or implementation within the organisations, and two of the four persons have experienced some degree of resistance to their international activities. For instance, one 232

233 participant, who herself holds the position of director, has met little support from the board and from most of the staff for her attempts to involve the school in European or Nordic collaboration activities. The overall picture seems to be that the Grundtvig in-service training grants result in significant profits at individual level (and partly among peers), but low profits at organizational and institutional level. In sum, the effectiveness of this action in Denmark can be characterised as somewhere between high and average (though of course the concept of average has limited meaning when we lack a basis for comparison). It seems that courses generally are used for relevant purposes by a relevant mix of beneficiaries, but that the dissemination of knowledge and inspiration from the course participants to their environments could have been better. 4. EFFICIENCY 4.1. Is the clarity and stability of the overall legal/formal framework of programme management and control system sufficient (at both the National Agency and the beneficiaries level)? The perception of this differs somewhat according to the position of the informants. The official in the Danish Agency and also the chairman of the national association for popular adult education indicate that no ambiguity or instability of the overall legal/formal framework of programme management and control system has been evident, neither at the national nor at the beneficiaries level. In this sense the Commission is fulfilling its guiding role in the implementation process. The individual beneficiaries are not strongly dissatisfied with the management and control procedures; but they find that especially in the application process questions often arise and they have to contact the agency for clarification. One informant says that assistance from the national agency is easy for him to get, because he lives in Copenhagen, which enables him to show up in person without too much effort, but this kind of support on applying could be difficult for an employee in a small adult school in the opposite end of the country to get. By this the informant indicates that the application form may raise questions that need faceto-face communication with the national agency to be clarified; e.g. what and where in the form to describe specific elements of a training course. Another individual beneficial supports this picture by recommending future appliers of Grundtvig programme participation to attend information meetings held by the national agency. According to her information meetings provide an opportunity to ask questions and get direct answers from the agency on the application process. Furthermore, these meetings work as an opportunity for appliers to discuss courses, program participation, application procedure etc. with each other. In other words to share questions and knowledge with people in the same situation as them self (individual adult education teachers, who want to participate in Grundtvig LLP). In general the application process would gain from being proceeded in native language (Danish) alone. This procedural change is being recommended by several individual beneficiaries for a smoother, more user oriented and less bureaucratic process Adequacy of management resources. Do the national stakeholders (the NA, institutional beneficiaries) possess adequate resources to ensure sound management of the programme? What are they key areas of shortages, if any? Do they use external assistance (external experts, contractors) to perform some of the functions? No one (neither the national agency nor institutional beneficiaries) has directly formulated difficulties in management of resources. Experiences from individual beneficiaries, however, indicate that they have to handle questions on the application process by them self and without the necessary support from own institution. It indicates that institutional beneficiaries could provide further resources to support their programme applying employees. Furthermore, neither the national agency nor institutional beneficiaries mentions how they use external assistance to perform some functions Is the monitoring information used for management purposes? What would be other examples of its use besides putting it into National Agency s reports to National Authority and European Commission? According to the official at the National Agency, monitoring information (for instance, evaluation reports of individual beneficiaries) is not used systematically for improvements of the administration process. In this sense controlling procedures like the written evaluation reports from individual beneficiaries seem to be 233

234 performed more as summative course evaluation reflecting accountability to participants, National Authorities and European Commission than an improvement-oriented formative evaluation. A bias of this evaluation procedure is, thus, that the evaluation questions concern elements of the single course, which the National Agency, which is responsible for the evaluation reports, has no influence on (like course material, didactic approach in the course, organisation of the course etc.). Feedback on such issues should be given to the specific course provider. However, to let the National Agency be responsible for this feedback process between participants and course providers is assumed to be a task taking up a significant amount of resources. In light of this it would make sense to have participants of each course to fill out an evaluation report before they leave the course and the course provider. A procedure like that would increase the possibility that the course provider achieves the relevant feedback from participants and thus be able to improve organisation, content, didactic approaches and teaching materials for future course activities. Making the reporting procedures more meaningful seems to rely on a two-step evaluation procedure; taking a step 1 as an evaluation to the course provider and step 2 as an evaluation for the national agency. While the first step focuses on participants experience with the specific course, the second focuses on general application procedural elements, legal framework, resource management, barriers on access and applying as well as dissemination of possibilities within the framework of Grundtvig programmes and the European Commission s efforts on lifelong learning. 234

235 ANNEX 4. LIST OF INTERVIEWS A. EU level interviews No. Institution Type Name, surname and position of interviewee Date and type 1. European Commission, DG EAC EU level authorities Teresa Echevarria, LLP programme administration Programme Manager EU policies Coordination des actions de valorisation et diffusion 21/05/2010 Face to face 2. European Commission, DG EAC EU level authorities Filip Van Depoele, Deputy Head of Unit 25/05/2010 Face to face European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC European Commission, DG EAC EU level authorities Pedro Chaves, Principal Administrator 21/05/2010 EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities EU level authorities 20. EACEA EU level authorities Luca Dalpozzo, LLP coordination Jose Pessanha, Oversight of the Education & Training 2010/ Lisbon process Jean-Yves Stefani, KA 1 Policy cooperation and innovation / studies & comparative research Alenka Kampl, Head of Sector, Coordination of National Agencies Youri Devuyst, Principal Administrator, Jean Monnet Programme key activity 1: Jean Monnet Action Youri Devuyst (Principal Administrator, Jean Monnet Programme key activity 1: Jean Monnet Action) with Bernaldo de Quiros, Head of Unit in attendance Renato Girelli and Youri Devuyst Carlos Brieva Collado, LLP link Margarita Lago, Deputy Head of Unit, Comenius coordinator Alan Smith, Grundtvig Coordinator and Deputy Head of Unit EAC-B4 Elena Pascual-Jimenez, Administrator, Jean Monnet Programme - key activity 2: Specified Institutions and ETF Godelieve van den Brande, responsible for KA3 within the Transversal Programme of the LLP Klara Engels-Perenyi, Desk Officer Kateřina Kapounová Bavorová, Administrator Isabelle Albertone, Head of Sector, LLP programme administration Géraldine Libreau, Grundtvig Programme and Thematic Assistant Ute Haller-Block, Head of Unit Lifelong Learning: Leonardo da Vinci, Grundtvig and Dissemination. Before 1 September she was deputy head of Unit of 21/06/2010 Face to Face 18/04/10 Face to Face 18/05/2010 Face to Face 13/05/ /06/2010 Face to Face 06/05/ /05/10 Telephone 29/10/2010 Face to Face 12/05/ /06/2010 Face to Face 22/06/2010 Face to Face 16/11/2010 Face to Face 06/09/2010 Telephone 27/08/2010, Telephone 29/10/2010 Face to Face 01/09/2010, Telephone 02/09/2010, Telephone 22/06/2010 Face to Face 16/12/2010 Telephone 02/06/2010 Face to Face 16/11/

236 No. Institution Type Name, surname and position of interviewee Date and type Vocational training, Leonardo da Vinci programme 21. EACEA EU level authorities Roberta Manoni, Programme Manager KA4 22. EACEA EU level authorities Maria Luisa Garcia Minguez Giles Gervais Katja De Sousa Face to Face 19/05/2010 Face to Face 25/05/2010 Face to Face 23. EACEA EU level authorities Gillian McLaughlin, Acting Head of Unit 21/05/ EACEA EU level authorities Brian Holmes, Head of LLP Department 25. EACEA EU level authorities Ralf Rahders, Head of Unit 26. EACEA EU level authorities Adam Pokorny, LLP Comenius 27. Committee of the Regions EU level authorities 28. UEAPME EU level social partners 29. Pendragon Language Teaching 30. E.N.T.E.R. Beneficiary (KA4) Klaus Boele, Administrator of the Commission for Culture, Education and Research Helen Hoffmann, Representative of the UEAPME in the LLP Committee 14/04/2010 Face to Face 14/04/2010 Face to Face 21/06/2010 Face to Face 03/06/10 Telephone 15/06/10 Telephone Beneficiary/Expert Carl Taylor, Director 20/05/2010 Petra Kampf, Project manager 28/05/2010 Telephone 31. Languages meet sport Beneficiary/Expert Ariella Cuk, Project leader 27/05/ MEDEA Beneficiary (KA4) Sally Reynolds, Project manager 33. European Schoolnet 34. University of Edinburgh 35. London School of Economics Project partner / Beneficiary (KA4) Beneficiary (Jean Monnet) Beneficiary (Jean Monnet) 36. LSE Institutional Beneficiary 37. LSE Institutional Beneficiary 38. European University Institute Beneficiary (Jean Monnet) Marc Durando, Executive Director 18/05/2010 Telephone 19/05/2010 Face to Face Prof. David Howarth 10/05/2010 Prof. David Chalmers 08/06/2010 Maurice Fraser, Senior Fellow in European Politics; Programme Director, LSE and Sciences Po European Double Masters Degree; Director, LSE European Public Lecture Series Vivian Winterhoof, Manager of the European Institute Telephone, 06/07/10 Telephone, 05/07/10 Marco Del Panta, Secretary General of the EUI 10/09/2010 B. National level interviews No. Institution Type Name, surname and position of interviewee Date and type Kaunas University of Technology, (Lithuania) Universidade Catolica Portuguesa - Escuela Superior de Biotecnologia; Instituto Politecnico de Coimbra (Porto, Portugal) Directorate General Science and Technology of Portugal (Lisbon, Portugal) Institutional beneficiary (partner) Institutional beneficiaries (coordinator and partner) National authorities Dr. Rimantas Venskutonis Dr. Cristina Silva Dr. Rui Costa Magda Noronha Nuno Afonso Ana Mateus Vera Palma 13/05/2010 Telephone 26/05/2010 Face to face 27/05/2010 Face to face 236

237 No. Institution Type Name, surname and position of interviewee Date and type 4. NAEP (Slovakia) National agency 5. NAEP (Slovakia) National agency 6. NAEP (Slovakia) National agency 7. NAEP (Slovakia) National agency 8. MoEYS (Slovakia) National authority Charles University in Prague (Czech Republic) Faculty of Social Sciences Charles University in Prague (Czech Republic) Faculty of Education Charles University in Prague (Czech Republic) Institutional beneficiary Institutional beneficiary Institutional beneficiary 12. CHES (Czech Republic) Social partner 13. CHES (Czech Republic) Social partner Agenzia Nazionale per lo Sviluppo dell Autonomia Scolastica (ex INDIRE) Agenzia Nazionale per lo Sviluppo dell Autonomia Scolastica (ex INDIRE) MIUR Ministero dell Istruzione, Università e Ricerca (Italy) Liceo Scientifico Colombo Marigliano (NA) (Italy) Istituto Comprensivo Carbonera Carbonera (TV) (Italy) Direzione Didattica D.Lorenzo Milani 9 Circolo Didatico Sassari (SS) (Italy) ISS Palmeri Termini Imerese (PA) (Italy) Nucleo Territoriale Lombardia ANSAS (ex IRRE Lombardia) (Italy) Human Resource Development Center (Bulgaria) LLP National Agency for Comenius programme LLP National Agency for Comenius programme Ministry of education National Authority Institutional Beneficiary Upper secondary School Institutional Beneficiary Primary, lower secondary School Institutional Beneficiary Preprimary, primary, lower secondary School Institutional Beneficiary Upper Secondary School Local regional office of the LLP NA National agency Dana Petrová Erasmus - Programme Manager Barbora Hašková Erasmus - Mobility Petra Emílie Krňanská Erasmus - Mobility Olga Salingerová Erasmus - Control Karolina Gondková HE Department Michal Maršálek European Office LLP/Erasmus Institutional Coordinator Cyril Šimsa Faculty Coordinator Stanislava Vlčková Faculty Coordinator Vladimír Roskovec Deputy Director Libor Prudký Research Department Dr. Sara Pagliai Chief of Unit Mobility and Partnership Dr. Fiora Imberciadori Coordinator of the Italian LLP National Agency Dr. Donatella Amatucci Chief of Unit II of DG for international affairs (MIUR) Prof. Speranza D Alessandro Responsible for Comenius Partnership Prof. Marina Prete Responsible for Comenius Partnership Prof. Rita Paola Spanedda Head of School Prof. Giacomina Masia Responsible for Comenius Partnership Prof. Annunciata Speranza Responsible for Comenius Partnership Dr. Cristina Dichirico LLP Responsible for Lombardy region Margarita Djoganova Head of the Department 7/04/2010 and 12/04/2010 Face to face 10/05/2010, 12/05/2010 and 18/06/2010 Face to face, Telephone, e- mail 10/04/2010 and 12/04/2010 Face to face 12/05/2010, 28/05/2010 and 18/6/2010 Face to face and Telephone 13/04/2010 Telephone 13/04/2010 and 28/04/2010 Telephone 11/04/2010 Face to face 11/04/2010 Face to face 13/04/2010 Telephone 11/04/2010 Face to face 11/05/ /05/ /05/ /05/ /05/ /05/ /05/ /06/ /05/2010 Face to face 237

238 No. Institution Type Name, surname and position of interviewee Date and type Human Resource Development Center (Bulgaria) Human Resource Development Center (Bulgaria) Human Resource Development Center (Bulgaria) National Agency for VET (Bulgaria) Kindergarden Shumen (Bulgaria) 28. Svetlana Dimitrova (Bulgaria) 29. Svetlana Mihajlova (Bulgaria) 30. Krasimira Kostova (Bulgaria) University of Leipzig; WILMA; Local ERASMUS Student Initiatives (LEI) (Germany) University of applied science Nordhausen (International Office) (Germany) University of Halle (International Office) (Germany) National agency National agency National agency national authorities institutional beneficiary individual beneficiary individual beneficiary individual beneficiary institutional beneficiary / social partner national authorities institutional beneficiary / social partner Marchela Miteva collaborator programme Comenius Krasimira Karaasenova Department Analysis Natalia Kalandarova Head of the Department Analysis Deyan Pushkarov President of NAVET Veneta Pavlova contact person Svetlana Dimitrova individual assistant individual assistant individual assistant 4 students (Lars-Peter, Christiane, Christian F. and Micha) Thomas Hoffmann, National ERASMUS Expert Gritt Eisenkopf, Friederike Lange and Anna Emelyanova, ERASMUS office 34. DAAD (Germany) national agency Siegbert Wuttig, Director of the National agency University of Marburg (International Office) (Germany) Norwegian Centre for International Cooperation in Higher Education (SIU), Bergen (Norway) Volda University College, Volda (Norway) Norwegian Oil Industry Association (OLF), Stavanger (Norway) Norwegian Academy of Music, Oslo (Norway) Oslo University College (HiO) (Norway) 41. University of Oslo (Norway) 42. Ministry of Education and Research, Oslo (Norway) national authorities National Agency Institutional beneficiary Social partner Institutional beneficiary Institutional beneficiary Institutional beneficiary National authority 43. SIU, Bergen (Norway) National Agency Christopher Moss, National ERASMUS Expert Cecilie Høisæter, Contact person and Coordinator Erasmus, Unit for Higher Education and Nordic Coordination Stian Thowsen, Adviser, Unit for Higher Education and Nordic Coordination Cecilie Wilhelmsen, Head of International office Thina Hagen, Leader competence department at OLF and Member of the National Programme Committee Erasmus , representative for the Confederation of Norwegian Enterprise (NHO) Knut Myhre, Senior Adviser International office Morten Gransæther Kielland, Senior Adviser International office Heide Remmele Hindahl, Adviser International office Tron Trondal, International Coordinator at the Faculty of Social Sciences Gro Beate Vige, Senior Adviser at the Department of Higher Education and Member of the National Programme Committee Erasmus Kari Omdahl, Higher Executive Officer, Unit for Higher Education and Nordic Coordination Cecilie Høisæter, Contact person and Coordinator Erasmus, Unit for Higher Education and Nordic 04/05/2010 Face to face 25/05/2010 Face to face 10/05/2010 Face to face 11/05/2010 Face to face 10/05/2010 Telephone 12/ Telephone 04/05/2010 Face to face 10/05/2010 Telephone 03/05/10 Face to Face 04/05/10 Telephone 04/05/10 face to face 05/05/10 Telephone 17/05/10 Telephone 29/04/2010 Telephone 03/05/2010 Telephone 05/05/2010 Telephone 06/05/2010 Face to face 06/05/2010 Face to face 07/05/2010 Face to face 10/05/2010 Face to face 10/05/2010 Telephone 238

239 No. Institution Type Name, surname and position of interviewee Date and type Coordination Union of Education Norway, Oslo (Norway) Norwegian Society of Graduate Technical and Scientific Professionals (Tekna), Oslo (Norway) Education Exchanges Support Foundation (EESF) (Lithuania) Social partner Social partner National Agency Tove Hauge, Member of the National Programme Committee Erasmus (since 2005, since 2009: deputy), representative for the Norwegian Confederation of Unions for Professionals (Unio) Atle Tangedal, Head of department Teknakurs, Member of the National Programme Committee Erasmus , representative for the Federation of Norwegian Professional Associations (Akademikerne) Ilona Kazlauskaitė, Head of the Academic Exchanges Unit Arūnė Karosaitė, Academic Exchanges Unit 47. EESF (Lithuania) National Agency Dovilė Kavaliauskaitė, Director Deputy 48. Vilnius University (Lithuania) Lithuanian Music and Theatre Academy (Lithuania) Vilnius Gediminas Technical University (Lithuania) Ministry of Education and Science (Lithuania) Ministry of Education and Science (Lithuania) Institutional beneficiary Institutional beneficiary Institutional beneficiary National authorities National authorities 53. EESF (Lithuania) National authorities Danish Agency for International Education, Ministry of Education (Denmark) Vifin, Centre for Knowledge on integration (Denmark) Vifin, Centre for Knowledge on integration (Denmark) Sports day high school, Fredericia (Denmark) Sports day high school, Fredericia (Denmark) Sports day high school, Fredericia (Denmark) Center for General Adult Education Nordsjaelland (Denmark) Centre for General Adult Education, Vestsjaelland (Denmark) Chairman of National Association of Popular Adult Education (Denmark) Ministry of Education and Sport (Slovenia) National authority Individual beneficiary Representing organisation with individual beneficiary Individual beneficiary Representing organisation with individual beneficiary Representing organisation with individual beneficiary Individual beneficiary Individual beneficiary Social partner National authority Raimonda Markevičienė, Head of International Relations Unit Aistė Simelavičienė, International Relations Unit Rima Rimšaitė, LLP/Erasmus institutional coordinator Lilija Gončarovienė, LLP/Erasmus institutional coordinator, International Office Virginija Rimkevičienė Head of Academic Mobility Unit Jurgita Strumskienė, Head of the International Cooperation Unit (to be conducted) Ilona Kazlauskaitė, Head of the Academic Exchanges Unit Annemarie Holm, International advisor Herluf Schelde, teacher and consultant Ib Jespersgaard, Director Inge Skafsgaard Michael Schmidt Dorte Pedersen Jesper Thing-Simonsen Tove Aakjaer Jacobsen Per Paludan Hansen Andreja Čufer, head of the office 12/05/2010 Telephone 12/05/2010 Telephone 31/03/2010 Face to face 31/03/2010 Face to face 07/04/2010 Face to face 08/04/2010 Face to face 12/04/2010 Face to face 12/04/2010 Face to face 21/04/2010 Face to face 20/04/2010 Telephone May 11/ 2010, face-to-face May 20/ 2010, Telephone May 25 /2010, Telephone May 20/ 2010, Telephone May 21/ 2010, Telephone May 21/ 2010, Telephone May 20 /2010, Telephone May 26/ 2010, Telephone May 21 /2010, Telephone Face to face 17/05/

240 No. Institution Type Name, surname and position of interviewee Date and type Ministry of Education and Sport (Slovenia) Ministry of Education and Sport (Slovenia) National authority National authority 66. CMEPIUS (Slovenia) National agency Klavdija Tratar, office for European matters Ema Perme, adult education Maja Flander, deputy director of NA, responsible for LLP 67. CMEPIUS (Slovenia) National agency Simona Mehle, Grundtvig coordinator 68. CMEPIUS (Slovenia) National agency Urška Slapšak, Grundtvig coordinator 69. Taka tuka (Slovenia) 70. Taka tuka (Slovenia) 71. LU Jesenice (Slovenia) 72. Univerza za 3. življenjsko obdobje (Slovenia) Institutional beneficiary Institutional beneficiary Institutional beneficiary Institutional beneficiary 73. CINOP (the Netherlands) National Agency Anne Potters 74. CINOP (the Netherlands) National Agency Peter Luijten MBO-raad (VET sector) (the Netherlands) Deltion College (the Netherlands) ROC van Amsterdam (the Netherlands) Kennis Centrum Handel (the Netherlands) Nova college (the Netherlands) ROC Landstede (the Netherlands) Umbrella organisation Beneficiary (Leonardo da Vinci) Beneficiary (Leonardo da Vinci) Beneficiary (Leonardo da Vinci) Beneficiary (Leonardo da Vinci) Beneficiary (Leonardo da Vinci) Veronika Gaber Korbar, project leader, Katarina Picelj, project koordinator Maja Radinovič Hajdič, director Magdalena Ažman, project coordinator Manfred Polzin Herman Zwakenberg Marian Markenhof Pasqualino Mare Johan Sloesen Udo Lut Face to face 17/05/2010 Face to face 17/05/2010 Face to face 11/05/2010 Face to face 11/05/2010 Face to face 07/05/2010 Face to face 10/04/2010 Face to face 10/04/2010 Face to face 11/05/2010 Face to face 12/05/ /03/2010 Face-to-face, (02/06/2010 Telephone) 15/03/2010 Face-to-face 22/04/2010, 31/05/2010 Telephone 06/04/2010, 31/05/2010 Telephone 14/04/2010, 01/06/2010 Telephone 16/04/2010, 31/05/2010 Telephone 31/05/2010 Telephone 31/05/2010 Telephone 81. Freie Universität Berlin Beneficiary/Expert Prof. Wolfgang Mackiewicz 20/05/

241 ANNEX 5. SURVEY RESULTS 5.1. Survey of the National Authorities and National Agencies involved in the implementation of the LLP 1. In your opinion, are the administrative arrangements (such as project administration procedures & document templates, eligibility requirements for expenditure and similar) for similar types of actions (i.e. mobility, partnership/network, other) well harmonised and unified across different LLP sub-programmes? (NATIONAL AGENCIES only) The administrative arrangements for similar types of actions shall be further harmonised across different LLP sub-programmes to achieve full uniformity Frequency 15 62,5% No change is needed, the current level of harmonisation shall be maintained 9 37,5% The administrative arrangements shall be further differentiated to better reflect specificities of the LLP sub-programmes 0 0% Total % 2. Please estimate which share of the total budget of your institution allocated to LLP management is covered by the NA operating grant (i.e., the estimated share = NA operating grant / the total budget of your institution allocated to LLP management). (NATIONAL AGENCIES only) Frequency 2007 Up to 50% 12 57,1% % 5 23,8% 66-80% 1 4,8% 81-90% 0 0% % 3 14,3% 2008 Up to 50% 12 57,1% % 5 23,8% 66-80% 2 9,1% 81-90% 0 0% % 3 14,3% 2009 Up to 50% 12 54,5% % 5 22,7% 66-80% 2 9,1% 81-90% 1 4,5% % 2 9,1% 241

242 3. Please provide your assessment of the following aspects of European Commission guidance (guides to NAs, LLP guides, application and reporting forms, various templates, and other guidance) on LLP management, monitoring and control issues during initial years of the LLP and more recently, at present. (NATIONAL AGENCIES only) During initial years of the LLP ( ) More recently, at present (2009 and later) Frequency Frequency A. Timely ex-ante formulation and adoption/setting of rules and administrative arrangements Very positive 0 0% 3 11,5% Rather positive 6 23,1% 16 61,5% Rather negative 14 53,8% 6 23,1% Very negative 6 23,1% 1 3,8% Do not know / can not answer 0 0% 0 0% B. Stability and continuity of rules and administrative arrangements Very positive 0 0% 3 11,5% Rather positive 6 23,1% 15 57,7% Rather negative 14 53,8% 6 23,1% Very negative 6 23,1% 2 7,7% Do not know / can not answer 0 0% 0 0% C. Adequacy of training support related to the introduction of new rules and administrative arrangements Very positive 0 0% 2 7,7% Rather positive 6 23,1% 17 65,4% Rather negative 15 57,7% 5 19,2% Very negative 4 15,4% 1 3,8% Do not know / can not answer 1 3,8% 1 3,8% D. Availability of ad hoc guidance (ability to react to ad hoc requests for guidance in due time) Very positive 1 3,8% 13 50,0% Rather positive 19 73,1% 10 38,5% Rather negative 4 15,4% 2 7,7% Very negative 1 3,8% 0 0% Do not know / can not answer 1 3,8% 1 3,8% Total % % 4. Even though the Guide to NAs clearly states that The NA has no formal responsibility with regard to the management of the centralised actions of the programme, which falls in the scope of competence of the Executive Agency, the co-ordination between centralised and decentralised actions and therefore between EACEA and NAs is perceived to be important. Please assess your co-operation experience with the Executive Agency. (NATIONAL AGENCIES only) Frequency Current level of co-operation is sufficient 3 11,5% Co-operation with the Executive Agency shall be strengthened 23 88,5% Total % 5. If the co-operation shall be strengthened, please indicate the possible areas of co-operation. (NATIONAL AGENCIES only) Frequency Information/promotion of the centralised actions Very important 13 56,5% 242

243 Important 9 39,1% Somewhat important 1 4,3% Valorisation of project results implemented under the centralised actions Very important 18 78,3% Important 4 17,4% Somewhat important 1 4,3% Monitoring and support to grant beneficiaries of the centralised actions Very important 3 13% Important 8 34,8% Somewhat important 12 52,2% 6. Please indicate the main obstacles which impede adequate co-operation with the Executive Agency. (NATIONAL AGENCIES only) Frequency NAs formally have no responsibilities related to the co-ordination of centralised actions and co-operation with the Executive Agency Very important 13 52% Important 8 32% Somewhat important 4 16% No funding for NAs related to co-operation in the implementation of the centralised actions Very important 17 65,5% Important 6 23,1% Somewhat important 3 11,5% Difficulties in collaboration with the Executive Agency Very important 9 37,5% Important 6 25% Somewhat important 9 37,5% 7. In your opinion do the current management and control procedures applicable to grant beneficiaries provide for proper and correct implementation of the LLP decentralised actions? Current management and control procedures (NATIONAL AGENCIES only) Frequency provide for proper and correct implementation of the programme 11 42,3% shall be further enhanced (e.g., requirements shall be made more detailed, precise) in order to ensure proper and correct implementation of the programme 2 7,69% could be simplified to provide for more efficient implementation of the programme without no (substantial) negative effect on ensuring of proper and correct implementation of the programme 17 65,38% Total Please assess the relative administrative burden and costs of the following LLP monitoring, supervision and control mechanisms applied by the NAs. (NATIONAL AGENCIES only) The requirements are too demanding The requirements are well balanced Frequency Frequency 243

244 Project evaluation and selection procedures Project financial management and control procedures Procedures related to monitoring of the projects Requirements related to the dissemination and exploitation of projects and programme results Requirements related to the preparation of Annual Work Programmes of the NA Requirements related to reporting to the European Commission Procedures related to reporting to the National Authorities Administrative burden associated with controls carried out by the National Authorities 8 32% 17 68% 12 46,2% 14 53,8% 2 8% 23 92% 4 15,4% 22 84,6% 7 28% 18 72% 23 88,5% 3 11,5% 8 30,8% 18 69,2% 11 42,3% 15 57,7% 9. Please compare the relative administrative burden and costs of the current LLP monitoring, supervision and control mechanisms applied by the NAs in comparison to the pre-llp programmes: (NATIONAL AGENCIES only) Frequency Project evaluation and selection procedures The administrative burden/costs increased 12 46,2% No major change 11 42,3% The administrative burden/cost decreased 3 11,5% Project financial management and control procedures The administrative burden/costs increased 16 61,5% No major change 5 19,2% The administrative burden/cost decreased 5 19,2% Procedures related to monitoring of the projects The administrative burden/costs increased 10 38,5% No major change 15 57,7% The administrative burden/cost decreased 1 3,8% Requirements related to the dissemination and exploitation of projects and programme results The administrative burden/costs increased 17 65,4% No major change 9 34,6% The administrative burden/cost decreased 0 0% Requirements related to the preparation of annual work programmes of the NA The administrative burden/costs increased 10 38,5% No major change 8 30,8% The administrative burden/cost decreased 8 30,8% Requirements related to reporting to the European Commission The administrative burden/costs increased 18 69,2% No major change 8 30,8% The administrative burden/cost decreased 0 0% Procedures related to reporting to the National Authorities 244

245 The administrative burden/costs increased 17 68% No major change 8 32% The administrative burden/cost decreased 0 0% Administrative burden associated with controls carried out by the National Authorities The administrative burden/costs increased 23 88,5% No major change 3 11,5% The administrative burden/cost decreased 0 0% 10. Please choose one of the following statements concerning LLP financial management procedures (types of financing). (NATIONAL AGENCIES only) Frequency Current project financial management arrangements provide for proper balance between financing on the basis of real cost 4 and financing based on fixed cost (lump sums and flat rate 15,4% grants) Share of project financing based on fixed costs (lump sums and flat rate grants) shall be further increased 22 84,6% Share of project financing based on real costs shall be increased 0 0% Total % 11. Please assess the clarity of requirements (methodology) for the following primary checks executed by the NAs: (NATIONAL AGENCIES only) Frequency Analysis of final reports Very clear 15 57,7% Rather clear 10 38,5% Rather unclear 0 0% Very unclear 1 3,8% Do not know/ can not answer 0 0% Desk check of supporting material for costs claimed Very clear 7 26,9% Rather clear 16 61,5% Rather unclear 3 11,5% Very unclear 0 0% Do not know/ can not answer 0 0% On the spot check during the implementation of supported activity Very clear 3 11,5% Rather clear 11 42,3% Rather unclear 11 42,3% Very unclear 0 0% Do not know/ can not answer 1 3,8% On the spot check (audit) undertaken upon completion of a supported activity Very clear 4 15,4% Rather clear 17 65,4% Rather unclear 5 19,2% 245

246 Very unclear 0 0% Do not know/ can not answer 0 0% System audit of recurrent beneficiaries Very clear 5 19,2% Rather clear 16 61,5% Rather unclear 0 0% Very unclear 2 7,7% Do not know/ can not answer 3 11,5% Total % 12. Please provide your opinion on the requirements for sample (minimum percentages and numbers of grant agreements to be checked) of the following primary checks: (NATIONAL AGENCIES only) Frequency Analysis of final reports The sample is too broad 2 7,7% The sample is well balanced 24 92,3% The sample is too small 0 0% Desk check of supporting material for costs claimed The sample is too broad 5 19,2% The sample is well balanced 21 80,8% The sample is too small 0 0% On the spot check during the implementation of supported activities The sample is too broad 3 12% The sample is well balanced 22 88% The sample is too small 0 0% On the spot check (audit) undertaken upon completion of a supported activity The sample is too broad 2 8% The sample is well balanced 23 92% The sample is too small 0 0% System audit of recurrent beneficiaries The sample is too broad 5 20,8% The sample is well balanced 19 79,2% The sample is too small 0 0% 13. Please assess the relevance and actual usage of the monitoring information provided by the grant beneficiaries in their progress and final reports for your internal decision making needs (i.e. project control and monitoring); external reporting of achievements (i.e. reporting to EC, National Authority, reporting and promotion to wider public): (NATIONAL AGENCIES only) For your internal decision making needs (i.e. project control and monitoring) For external reporting of achievements (i.e. reporting to EC, National Authority, reporting and promotion to wider public) Frequency Frequency Information on organisational matters (travel, accommodation, learning/placement environment, difficulties encountered, sufficiency of the grant amount, etc.) 246

247 Actively used 19 73,1% 13 50% Sometimes used 6 23,1% 12 46,2% Content of the activities and outputs (descriptive part) Actively used 21 80,8% 18 69,2% Sometimes used 5 19,2% 8 30,8% Content of the activities and outputs (statistical data) Actively used 17 65,4% 21 80,8% Sometimes used 9 34,6% 5 19,2% Evaluation of results/impact/quality of the activities Actively used 18 69,2% 18 69,2% Sometimes used 8 30,8% 8 30,8% Information/recommendations on dissemination and exploitation of results/experience of the activities Actively used 16 64% 14 56% Sometimes used 9 36% 11 44% 14. Please assess your experience related to the following aspects of the functioning of the LLP Link: (NATIONAL AGENCIES only) At the beginning of use ( ) More recently (this year) Frequency Frequency Functionalities of the LLP Link related to the collection and evaluation of applications (application e-forms, etc.) Very positive 0 0% 4 15,4% Rather positive 5 19,2% 19 73,1% Rather negative 14 53,8% 2 7,7% Very negative 7 26,9% 1 3,8% Do not know / can not answer 0 0% 0 0% Functionalities of the LLP Link related to monitoring purposes (possibilities to generate standard reports, use of LLP data for analysis purposes, etc.) Very positive 1 3,8% 1 3,8% Rather positive 3 11,5% 13 50% Rather negative 8 30,8% 9 34,6% Very negative 13 50% 3 11,5% Do not know / can not answer 1 3,8% 0 0% User-friendliness of the LLP Link (automated input of data, utility of LLP link for management purposes, etc.) Very positive 0 0% 2 7,7% Rather positive 1 3,8% 17 65,4% Rather negative 16 61,5% 4 15,4% Very negative 8 30,8% 3 11,5% Do not know / can not answer 1 3,8% 0 0% Reliability of LLP Link data Very positive 0 0% 3 11,5% Rather positive 8 30,8% 14 53,8% Rather negative 10 38,5% 8 30,8% Very negative 7 26,9% 1 3,8% Do not know / can not answer 1 3,8% 0 0% Practical use of the LLP Link for the daily management of LLP programme and projects lifecycle Very positive 0 0% 2 7,7% 247

248 Rather positive 4 15,4% 13 50% Rather negative 13 50% 9 34,6% Very negative 8 30,8% 2 7,7% Do not know / can not answer 1 3,8% 0 0% Supporting material of the LLP Link users (manuals, handbooks, etc.) Very positive 1 3,8% 1 3,8% Rather positive 9 34,6% 20 76,9% Rather negative 10 38,5% 4 15,4% Very negative 4 15,4% 0 0% Do not know / can not answer 2 7,7% 1 3,8% Training support from the EC on the use of the LLP Link Very positive 0 0% 0 0% Rather positive 3 11,5% 13 50% Rather negative 14 53,8% 11 42,3% Very negative 8 30,8% 1 3,8% Do not know / can not answer 1 3,8% 1 3,8% Ad hoc support from the EC on the use of the LLP Link Very positive 2 7,7% 3 11,5% Rather positive 12 46,2% 16 61,5% Rather negative 7 26,9% 5 19,2% Very negative 2 7,7% 0 0% Do not know / can not answer 3 11,5% 2 7,7% Interoperability of the LLP Link with other IT systems, which you use internally for LLP management Very positive 0 0% 1 3,8% Rather positive 1 3,8% 10 38,5% Rather negative 6 23,1% 8 30,8% Very negative 14 53,8% 3 11,5% Do not know / can not answer 5 19,2% 4 15,4% Total % % 15. Please provide your opinion on the functioning of the valorisation platforms (EVE, EST, ADAM): (NATIONAL AGENCIES only) Frequency EST: Visibility and ease of access to project results for those seeking information Very positive 2 7,7% Rather positive 8 30,8% Rather negative 4 15,4% Very negative 2 7,7% Do not know / can not answer 10 38,5% EST: Adequacy of the content for valorisation purposes Very positive 3 11,5% Rather positive 9 34,6% Rather negative 7 26,9% Very negative 0 0% Do not know / can not answer 7 26,7% EST: User-friendliness of data input for project beneficiaries Very positive 1 3,8% 248

249 Rather positive 11 42,3% Rather negative 7 26,9% Very negative 2 7,7% Do not know / can not answer 5 19,2% ADAM: Visibility and ease of access to projects results for those seeking informatikon Very positive 9 34,6% Rather positive 8 30,8% Rather negative 3 11,5% Very negative 0 0% Do not know / can not answer 6 23,1% ADAM: Adequacy of the content for valorisation purposes Very positive 8 30,8% Rather positive 10 38,5% Rather negative 2 7,7% Very negative 0 0% Do not know / can not answer 6 23,1% ADAM: User-friendliness of data input for project beneficiaries Very positive 8 30,8% Rather positive 10 38,5% Rather negative 3 11,5% Very negative 0 0% Do not know / can not answer 5 19,2% EVE, EST, ADAM: Synergy between the LLP valorisation platforms EVE, EST and ADAM Very positive 0 0% Rather positive 8 30,8% Rather negative 5 19,2% Very negative 3 11,5% Do not know / can not answer 10 38,5% Total % 16. Please assess your experience related to the following aspects of the functioning of the present and previous information exchange and communication platforms (CIRCA, LLP NA Forum, Nety): (NATIONAL AGENCIES only) CIRCA: Overall user-friendliness of the system Frequency Very positive 0 0% Rather positive 17 65,4% Rather negative 9 34,6% Very negative 0 0% Do not know / can not answer 0 0% CIRCA: Functionalities related to the exchange of documents (guides to NAs and related documents, LLP Committee documents, etc.) Very positive 7 26,9% Rather positive 14 53,8% Rather negative 3 11,5% Very negative 1 3,8% Do not know / can not answer 1 3,8% 249

250 CIRCA: Functionalities related to communication (forums, possibilities to provide questions and comments on documents, etc.) Very positive 0 0% Rather positive 7 26,9% Rather negative 13 50% Very negative 5 19,2% Do not know / can not answer 1 3,8% LLP NA forum: Overall user-friendliness of the system Very positive 1 3,8% Rather positive 6 23,1% Rather negative 6 23,1% Very negative 3 11,5% Do not know / can not answer 10 38,5% LLP NA forum: Functionalities related to communication (possibilities to provide questions and comments, etc.) Very positive 1 3,8% Rather positive 7 26,9% Rather negative 5 19,2% Very negative 3 11,5% Do not know / can not answer 10 38,5% Nety: Overall user-friendliness of the system Very positive 6 23,1% Rather positive 7 26,9% Rather negative 4 15,4% Very negative 3 11,5% Do not know / can not answer 6 23,1% Nety: Functionalities related to the exchange of documents (guides to NAs and related documents, LLP Committee documents, etc.) Very positive 5 19,2% Rather positive 7 26,9% Rather negative 6 23,1% Very negative 2 7,7% Do not know / can not answer 6 23,1% Nety: Functionalities related to the communication (forums, possibilities to provide questions and comments on documents, etc.) Very positive 8 30,8% Rather positive 9 34,6% Rather negative 1 3,8% Very negative 2 7,7% Do not know / can not answer 6 23,1% Total % 17. What are the main challenges/difficulties related to secondary controls and declaration of assurance? (NATIONAL AUTHORITIES only) Frequency What are the main challenges/difficulties related to secondary controls and declaration of assurance? Difficulties related to the novelty of the procedure of declarations of assurance and secondary controls 250

251 Strongly agree 9 40,9% Rather agree 10 45,5% Rather disagree 2 9,1% Strongly disagree 0 0% Do not know / can not answer 1 4,5% What are the main challenges/difficulties related to secondary controls and declaration of assurance? Untimely (delayed) guidance on the secondary controls and declaration of assurance Strongly agree 10 45,5% Rather agree 7 31,8% Rather disagree 3 13,6% Strongly disagree 1 4,5% Do not know / can not answer 1 4,5% What are the main challenges/difficulties related to secondary controls and declaration of assurance? Insufficiently detailed guidance on the secondary controls and declaration of assurance (i.e. undefined requirements for the minimum scope and sample of secondary controls, etc.) Strongly agree 4 18,2% Rather agree 8 36,4% Rather disagree 7 31,8% Strongly disagree 1 4,5% Do not know / can not answer 2 9,1% What are the main challenges/difficulties related to secondary controls and declaration of assurance? Overly demanding requirements (i.e, the scope and sample of secondary controls is too broad) Strongly agree 9 40,9% Rather agree 12 54,5% Rather disagree 0 0% Strongly disagree 0 0% Do not know / can not answer 1 4,5% What are the main challenges/difficulties related to secondary controls and declaration of assurance? Difficulties in co-operation with the National Agencies, unacceptable quality of the National Agencies reports Strongly agree 0 0% Rather agree 0 0% Rather disagree 1 4,5% Strongly disagree 19 86,4% Do not know / can not answer 2 9,1% Total % 18. Overall, how relevant are the LLP objectives and actions to the national priorities and socio/economic needs? (NATIONAL AUTHORITIES only) Frequency All objectives and actions are highly relevant 12 54,5% The objectives are important, but they do not address the most pressing concerns 7 31,8% The objectives and actions are not important at all 0 0% Do not know / cannot answer 3 13,6% Total % 251

252 19. To what extent do you agree with the following statements? All National Agencies National Authorities Frequency Frequency Frequency There are too many LLP objectives; they should be streamlined Strongly agree 23 47,9% 12 46,2% 11 50% Rather agree 19 39,6% 9 34,6% 10 45,5% Rather disagree 5 10,4% 4 15,4% 1 4,5% Strongly disagree 0 0% 0 0% 0 0% Do not know /can not answer 1 2,1% 1 3,8% 0 0% Mobility actions are more important/relevant than partnerships and projects Strongly agree 4 8,3% 2 7,7% 2 9,1% Rather agree 11 22,9% 4 15,4% 7 31,8% Rather disagree 17 35,4% 11 42,3% 6 27,3% Strongly disagree 12 25,0% 7 26,9% 5 22,7% Do not know /can not answer 4 8,3% 2 7,7% 2 9,1% Partnerships and projects are more important/relevant than mobility actions Strongly agree 1 2,1% 0 0% 1 4,5% Rather agree 6 12,5% 1 3,8% 5 22,7% Rather disagree 26 54,2% 18 69,2% 8 36,4% Strongly disagree 9 18,8% 5 19,2% 4 18,2% Do not know /can not answer 6 12,5% 2 7,7% 4 18,2% Sectoral structure of the LLP reinforces the borders between different educational sectors and does not promote open learning environments Strongly agree 2 4,2% 1 3,8% 1 4,5% Rather agree 13 27,1% 7 26,9% 6 27,3% Rather disagree 20 41,7% 10 38,5% 10 45,5% Strongly disagree 12 25,0% 8 30,8% 4 18,2% Do not know /can not answer 1 2,1% 0 0% 1 4,5% Current structure of the LLP is appropriate for reaching different target groups Strongly agree 19 39,6% 11 42,3% 8 36,4% Rather agree 25 52,1% 13 50% 12 54,5% Rather disagree 4 8,3% 2 7,7% 2 9,1% Strongly disagree 0 0% 0 0% 0 0% Do not know /can not answer 0 0% 0 0% 0 0% The LLP cannot contribute to increased participation in lifelong learning, because it targets persons, who already participate in education and training Strongly agree 3 6,2% 1 3,8% 2 9,1% Rather agree 13 27,1% 6 23,1% 7 31,8% Rather disagree 11 22,9% 6 23,1% 5 22,7% Strongly disagree 15 31,2% 10 38,5% 5 22,7% Do not know /can not answer 6 12,5% 3 11,5% 3 13,6% Total % % % 20. If, hypothetically, the LLP and its sectoral programmes did not exist, what would be the level and quality of the following activities? All National Agencies National Authorities 252

253 Frequency Frequency Frequency Co-operation between the Participating Countries in the field of education policy Higher level and quality 0 0% 0 0% 0 0% Similar to current level and quality 5 10,4% 2 7,7% 3 13,6% Lower than current level and quality 39 81,2% 22 84,6% Do not know / cannot answer 4 8,4% 2 7,7% 2 9,1% Co-operation and interchange between providers of education and training Higher level and quality 1 2,1% 0 0% 1 4,5% 17 77,3% Similar to current level and quality 7 14,6% 4 15,4% 3 13,6% Lower than current level and quality 38 79,2% 21 80,8% 17 77,3% Do not know / cannot answer 2 4,2% 1 3,8% 1 4,5% Mobility of teachers and trainers Higher level and quality 2 4,2% 0 0% 2 9,1% Similar to current level and quality 0 0% 0 0% 0 0% Lower than current level and quality 43 89,6% 24 92,3% 19 86,4% Do not know / cannot answer 3 6,2% 2 7,7% 1 4,5% Mobility of students from primary, secondary and vocational schools, adult learners Higher level and quality 2 4,2% 0 0% 2 9,1% Similar to current level and quality 0 0% 0 0% 0 0% Lower than current level and quality 41 85,4% 22 84,6% 19 86,4% Do not know / cannot answer 5 10,4% 4 15,4% 1 4,5% Mobility of students from higher education institutions Higher level and quality 1 2,1% 0 0% 1 4,5% Similar to current level and quality 6 12,5% 2 7,7% 4 18,2% Lower than current level and quality 36 75,0% 20 76,9% 16 72,7% Do not know / cannot answer 5 10,4% 4 15,4% 1 4,5% Language learning in primary, secondary and vocational schools and of adult learners Higher level and quality 1 2,1% 0 0% 1 4,5% Similar to current level and quality 14 29,2% 8 30,8% 6 27,3% Lower than current level and quality 27 56,2% 15 57,7% 12 54,5% Do not know / cannot answer 6 12,5% 3 11,5% 3 13,6% Language learning in higher education institutions Higher level and quality 1 2,1% 0 0% 1 4,5% Similar to current level and quality 17 35,4% 9 34,6% 8 36,4% Lower than current level and quality 25 52,1% 15 57,7% 10 45,5% Do not know / cannot answer 5 10,4% 2 7,7% 3 13,6% Development of common EU-wide tools and methods for the recognition and evaluation of competences and skills Higher level and quality 0 0% 0 0% 0 0% Similar to current level and quality 5 10,4% 2 7,7% 3 13,6% Lower than current level and quality 39 81,2% 22 84,6% 17 77,3% Do not know / cannot answer 4 8,4% 2 7,7% 2 9,1% 253

254 Total % % % 21. To what extent do you agree with the following statements regarding the links between the LLP and national policies? All National Agencies National Authorities Frequency Frequency Frequency There are important synergies between the LLP and national policies Strongly agree 18 37,5% 9 34,6% 9 40,9% Rather agree 25 52,1% 14 53,8% 11 50,0% Rather disagree 3 6,2% 1 3,8% 2 9,1% Strongly disagree 1 2,1% 1 3,8% 0 0% Do not know /can not answer 1 2,1% 1 3,8% 0 0% The LLP duplicates national / European efforts (e.g. funding of European Social Fund or national funds for student mobility) Strongly agree 0 0% 0 0% 0 0% Rather agree 3 6,2% 2 7,7% 1 4,5% Rather disagree 21 43,8% 9 34,6% 12 54,5% Strongly disagree 23 47,9% 15 57,7% 8 36,4% Do not know /can not answer 1 2,1% 0 0% 1 4,5% The LLP is an important instrument for implementing national priorities (e.g. in the area of internationalisation) Strongly agree 21 43,8% 11 42,3% 10 45,5% Rather agree 22 45,8% 13 50% 9 40,9% Rather disagree 4 8,3% 1 3,8% 3 13,6% Strongly disagree 1 2,1% 1 3,8% 0 0% Do not know /can not answer 0 0% 0 0% 0 0% The LLP runs in parallel to national policies, but they do not interact (e.g. because national policy makers are not sufficiently involved in the LLP) Strongly agree 3 6,2% 3 11,5% 0 0% Rather agree 15 31,2% 8 30,8% 7 31,8% Rather disagree 22 45,8% 13 50% 9 40,9% Strongly disagree 8 16,7% 2 7,7% 6 27,3% Do not know /can not answer 0 0% 0 0% 0 0% The LLP supplements national policies by supporting actions that could not be supported by the national government (comparative studies, networks, etc.) Strongly agree 16 33,3% 8 30,8% 8 36,4% Rather agree 28 58,3% 16 61,5% 12 54,5% Rather disagree 1 2,1% 0 0% 1 4,5% Strongly disagree 2 4,2% 1 3,8% 1 4,5% Do not know /can not answer 1 2,1% 1 3,8% 0 0% Total % % % 22. To what extent do you agree with the following statements regarding synergies and overlaps between the sectoral programmes? All National Agencies National Authorities Frequency Frequency Frequency Integration of all sectoral programmes leads to the creation of additional outputs, which would not be developed, if the sectoral programmes were implemented separately 254

255 Strongly agree 11 22,9% 7 26,9% 4 18,2% Rather agree 22 45,8% 12 46,2% 10 45,5% Rather disagree 8 16,7% 4 15,4% 4 18,2% Strongly disagree 3 6,2% 3 11,5% 0 0% Do not know /can not answer 4 8,3% 0 0% 4 18,2% The synergies between sectoral programmes are limited to more effective administration and promotion of the LLP Strongly agree 6 12,5% 3 11,5% 3 13,6% Rather agree 23 47,9% 10 38,5% 13 59,1% Rather disagree 15 31,2% 11 42,3% 4 18,2% Strongly disagree 1 2,1% 1 3,8% 0 0% Do not know /can not answer 3 6,2% 1 3,8% 2 9,1% There are very important synergies between the centralised and decentralised actions of the LLP Strongly agree 0 0% 0 0% 0 0% Rather agree 10 20,8% 7 26,9% 3 13,6% Rather disagree 21 43,8% 10 38,5% 11 50% Strongly disagree 13 27,1% 8 30,8% 5 22,7% Do not know /can not answer 4 8,3% 1 3,8% 3 13,6% Overall, the synergies are small, and there is ample scope for improvement Strongly agree 10 20,8% 4 15,4% 6 27,3% Rather agree 21 43,8% 11 42,3% 10 45,5% Rather disagree 14 29,2% 9 34,6% 5 22,7% Strongly disagree 2 4,2% 2 7,7% 0 0% Do not know /can not answer 1 2,1% 0 0% 1 4,5% Further integration of the LLP and diminishing the borders between education sectors is the way forward to increase synergies Strongly agree 10 20,8% 6 23,1% 4 18,2% Rather agree 21 43,8% 9 34,6% 12 54,5% Rather disagree 10 20,8% 6 23,1% 4 18,2% Strongly disagree 4 8,3% 3 11,5% 1 4,5% Do not know /can not answer 3 6,2% 2 7,7% 1 4,5% The programmes and actions duplicate each other Strongly agree 0 0% 0 0% 0 0% Rather agree 6 12,5% 2 7,7% 4 18,2% Rather disagree 26 54,2% 15 57,7% 11 50% Strongly disagree 11 22,9% 7 26,9% 4 18,2% Do not know /can not answer 5 10,4% 2 7,7% 3 13,6% Important target groups are not covered by the LLP Strongly agree 1 2,1% 1 3,8% 0 0% Rather agree 8 16,7% 5 19,2% 3 13,6% Rather disagree 26 54,2% 11 42,3% 15 68,2% Strongly disagree 12 25% 8 30,8% 4 18,2% Do not know /can not answer 1 2,1% 1 3,8% 0 0% Total % % % 23. In your opinion, is the size of the LLP budget appropriate to what the Programme is set out to achieve? All National Agencies National Authorities Frequency Frequency Frequency 255

256 The size of the LLP budget is appropriate to achieving the objectives of the Programme Appropriate 3 6,2% 1 3,8% 2 9,1% Rather appropriate 4 8,3% 2 7,7% 2 9,1% Rather inappropriate 20 41,7% 10 38,5% 10 45,5% Inappropriate 20 41,7% 12 46,2% 8 36,4% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% The size of the Comenius budget is appropriate to achieving the quantified target of the Comenius Programme: at least 3 million pupils in joint education activities over the period Appropriate 1 2,1% 0 0% 1 4,5% Rather appropriate 9 18,8% 5 19,2% 4 18,8% Rather inappropriate 15 31,2% 6 23,1% 9 40,9% Inappropriate 16 33,3% 9 34,6% 7 31,8% Do not know / cannot answer 7 14,6% 6 23,1% 1 4,5% The size of the Erasmus budget is appropriate to achieving the quantified target of the Erasmus Programme: at least 3 million individuals in student mobility actions under the Erasmus and its predecessor programmes by 2012 Appropriate 1 2,1% 1 3,8% 0 0% Rather appropriate 11 22,9% 4 15,4% 7 31,8% Rather inappropriate 12 25% 6 23,1% 6 27,3% Inappropriate 18 37,5% 10 38,5% 8 36,5% Do not know / cannot answer 6 12,5% 5 19,2% 1 4,5% The size of the Leonardo da Vinci budget is appropriate to achieving the quantified target of the Leonardo da Vinci Programme: at least placements at enterprises per year by 2013 Appropriate 1 2,1% 0 0% 1 4,5% Rather appropriate 5 10,4% 3 11,5% 2 9,1% Rather inappropriate 19 39,6% 8 30,8% 11 50% Inappropriate 15 31,2% 9 34,6% 6 27,3% Do not know / cannot answer 8 16,7% 6 23,1% 2 9,1% The size of the Grundtvig budget is appropriate to achieving the quantified target of the Grundtvig Programme: mobility of at least 7,000 individuals involved in adult education per year by 2013 Appropriate 0 0% 0 0% 0 0% Rather appropriate 5 10,4% 3 11,5% 2 9,1% Rather inappropriate 14 29,2% 5 19,2% 9 40,9% Inappropriate 21 43,8% 12 46,2% 9 40,9% Do not know / cannot answer 8 16,7% 6 23,1% 2 9,1% Total % % % 24. Please assess proportionality of the financial allocations across the centralised and decentralised actions and sectoral programmes. All National Agencies National Authorities Frequency Frequency Frequency The LLP: The centralised actions Financial share should be increased 1 2,1% 1 3,8% 0 0% Financial share should be kept at the same level 11 22,9% 7 26,9% 4 18,2% Financial share should be decreased 30 62,5% 16 61,5% 14 63,6% Do not know / cannot answer 6 12,5% 2 7,7% 4 18,2% 256

257 The LLP: The decentralised actions Financial share should be increased 41 85,4% 21 80,8% 20 90,9% Financial share should be kept at the same level 4 8,3% 3 11,5% 1 4,5% Financial share should be decreased 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 2 4,2% 1 3,8% 1 4,5% The decentralised actions of the sectoral programmes: Mobility actions Financial share should be increased 27 56,2% 15 57,7% 12 54,4% Financial share should be kept at the same level 16 33,3% 8 30,8% 8 36,4% Financial share should be decreased 2 4,2% 1 3,8% 1 4,5% Do not know / cannot answer 3 6,2% 2 7,7% 1 4,5% The decentralised actions of the sectoral programmes: Partnership actions Financial share should be increased 17 35,4% 8 30,8% 9 40,9% Financial share should be kept at the same level 25 52,1% 13 50% 12 54,5% Financial share should be decreased 2 4,2% 2 7,7% 0 0% Do not know / cannot answer 4 8,3% 3 11,5% 1 4,5% The decentralised actions of the sectoral programmes: Other actions Financial share should be increased 9 18,8% 7 26,9% 2 9,1% Financial share should be kept at the same level 18 37,5% 7 26,9% 11 50% Financial share should be decreased 9 18,8% 3 11,5% 6 27,3% Do not know / cannot answer 12 25% 9 34,6% 3 13,6% The sectoral programmes: Comenius Financial share should be increased 24 50% 12 46,2% 12 54,5% Financial share should be kept at the same level 16 33,3% 9 34,6% 7 31,8% Financial share should be decreased 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 7 14,6% 4 15,4% 3 13,6% The sectoral programmes: Erasmus Financial share should be increased 22 45,8% 10 38,5% 12 54,5% Financial share should be kept at the same level 16 33,3% 9 34,6% 7 31,8% Financial share should be decreased 4 8,3% 3 11,5% 1 4,5% Do not know / cannot answer 6 12,5% 4 15,4% 2 9,1% The sectoral programmes: Leonardo da Vinci Financial share should be increased 26 54,2% 15 57,7% 11 50% Financial share should be kept at the same level 15 31,2% 7 26,9% 8 36,4% Financial share should be decreased 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 6 12,5% 3 11,5% 3 13,6% The sectoral programmes: Grundtvig Financial share should be increased 34 70,8% 19 73,1% 15 68,2% Financial share should be kept at the same level 9 18,8% 4 15,4% 5 22,7% Financial share should be decreased 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 4 8,3% 2 7,7% 2 9,1% Total % % % 257

258 25. To what extent has the integration of the Programme in terms of the Programme content (such as the inclusion of new mobility and partnership actions) as well as its management (such as harmonisation in the Programme documents and procedures) led to the following positive or negative effects? All National Agencies National Authorities Frequency Frequency Frequency Integration has allowed better reaching the quantified mobility targets of the Programme To a large extent 1 2,1% 0 0% 1 4,5% To a reasonable extent 18 37,5% 10 38,5% 8 36,4% To a small extent 17 35,4% 9 34,6% 8 36,4% Not at all 8 16,7% 5 19,2% 3 13,6% Do not know / cannot answer 4 8,3% 2 7,7% 2 9,1% Programme has become more flexible, better meeting the needs of the beneficiaries and the target groups across different educational sectors To a large extent 5 10,4% 3 11,5% 2 9,1% To a reasonable extent 24 50% 14 53,8% 10 45,5% To a small extent 17 35,4% 8 30,8% 9 40,9% Not at all 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Integration allowed better mainstreaming of the horizontal policies (especially multilingualism and equality between man and women) in the sub-programmes and actions To a large extent 6 12,5% 2 7,7% 4 18,2% To a reasonable extent 8 16,7% 7 26,9% 1 4,5% To a small extent 16 33,3% 8 30,8% 8 36,4% Not at all 9 18,8% 4 15,4% 5 22,7% Do not know / cannot answer 9 18,8% 5 19,2% 4 18,2% The Programme has become more complex, making it necessary to manage a total of about 70 different actions To a large extent 25 52,1% 15 57,7% 10 45,5% To a reasonable extent 13 27,1% 6 23,1% 7 31,8% To a small extent 7 14,6% 4 15,4% 3 13,6% Not at all 2 4,2% 1 3,8% 1 4,5% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% The Programme integration resulted in more efficient promotion and information campaigns, higher awareness of the Programme benefits To a large extent 16 33,3% 11 42,3% 5 22,7% To a reasonable extent 20 41,7% 10 38,5% 10 45,5% To a small extent 11 22,9% 4 15,4% 7 31,8% Not at all 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 0 0% 0 0% 0 0% The Programme integration allowed for more effective dissemination and exploitation of project results To a large extent 14 29,2% 10 38,5% 4 18,2% To a reasonable extent 16 33,3% 7 26,9% 9 40,9% To a small extent 15 31,2% 8 30,8% 7 31,8% Not at all 2 4,2% 1 3,8% 1 4,5% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Integration allowed achieving better management links between the National Authorities (responsible for education and training policy) and the National Agencies (responsible for the implementation of the sub-programmes) To a large extent 12 25% 6 23,1% 6 27,3% 258

259 To a reasonable extent 18 37,5% 10 38,5% 8 36,4% To a small extent 9 18,8% 4 15,4% 5 22,7% Not at all 8 16,7% 5 19,2% 3 13,6% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% More synergy effects between the different sub-programmes and actions have occurred as a result of the Programme integration To a large extent 7 14,6% 5 19,2% 2 9,1% To a reasonable extent 21 43,8% 11 42,3% 10 45,5% To a small extent 19 39,6% 9 34,6% 10 45,5% Not at all 1 2,1% 1 3,8% 0 0% Do not know / cannot answer 0 0% 0 0% 0 0% The Programme integration increased administrative workload from the introduction of new procedures and documents To a large extent 20 41,7% 11 42,3% 9 40,9% To a reasonable extent 17 35,4% 7 26,9% 10 45,5% To a small extent 8 16,7% 5 19,2% 3 13,6% Not at all 2 4,2% 2 7,7% 0 0% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% There has been administrative simplification due to standardisation of procedures and documents To a large extent 0 0% 0 0% 0 0% To a reasonable extent 20 41,7% 11 42,3% 9 40,9% To a small extent 19 39,6% 11 42,3% 8 36,4% Not at all 8 16,7% 4 15,4% 4 18,2% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Total % % % 26. How should the integration of the LLP Programme be pursued in the future? All National Agencies National Authorities Frequency Frequency Frequency Integrating some Programme-specific objectives and reducing their number Strongly agree 32 66,7% 15 57,7% 17 77,3% Rather agree 14 29,2% 10 38,5% 4 18,2% Rather disagree 1 2,1% 1 3,8% 0 0% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Introducing common types of actions across the sub-programmes Strongly agree 24 50% 16 61,5% 8 36,4% Rather agree 19 39,6% 6 23,1% 13 59,1% Rather disagree 2 4,2% 2 7,7% 0 0% Strongly disagree 3 6,2% 2 7,7% 1 4,5% Do not know / cannot answer 0 0% 0 0% 0 0% Reducing the number of actions within the sub-programmes Strongly agree 18 37,5% 10 38,5% 8 36,4% Rather agree 21 43,8% 12 46,2% 9 40,9% Rather disagree 5 10,4% 2 7,7% 3 13,6% Strongly disagree 2 4,2% 2 7,7% 0 0% Do not know / cannot answer 2 4,2% 0 0% 2 9,1% Expanding the range of potential applicants that can apply for different sub-programmes and actions 259

260 Strongly agree 13 27,1% 7 26,9% 6 27,3% Rather agree 20 41,7% 13 50% 7 31,8% Rather disagree 14 29,2% 6 23,1% 8 36,4% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Involving new applicants (e.g. companies or regional authorities) in the Programme implementation Strongly agree 16 33,3% 9 34,6% 7 31,8% Rather agree 17 35,4% 11 42,3% 6 27,3% Rather disagree 7 14,6% 4 15,4% 3 13,6% Strongly disagree 4 8,3% 1 3,8% 3 13,6% Do not know / cannot answer 4 8,3% 1 3,8% 3 13,6% Improving co-operation between the Executive Agency (responsible for the centralised actions) and the National Agencies (responsible for the decentralised actions) Strongly agree 29 60,4% 18 69,2% 11 50% Rather agree 17 35,4% 8 30,8% 9 40,9% Rather disagree 0 0% 0 0% 0 0% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 2 4,2% 0 0% 2 9,1% Establishing a single National Agency responsible for all sub-programmes at the national level Strongly agree 21 43,8% 11 42,3% 10 45,5% Rather agree 8 16,7% 4 15,4% 4 18,2% Rather disagree 3 6,2% 1 3,8% 2 9,1% Strongly disagree 5 10,4% 4 15,4% 1 4,5% Do not know / cannot answer 11 22,9% 6 23,1% 5 22,7% Further harmonisation/standardisation in the Programme documents and procedures Strongly agree 28 58,3% 13 50% 15 68,2% Rather agree 17 35,4% 11 42,3% 6 27,3% Rather disagree 0 0% 0 0% 0 0% Strongly disagree 2 4,2% 2 7,7% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Exchanging good practices across the sub-programmes and actions Strongly agree 33 68,8% 17 65,4% 16 72,7% Rather agree 14 29,2% 9 34,6% 5 22,7% Rather disagree 0 0% 0 0% 0 0% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% More flexibility in the re-allocation of financial resources inside the Programme during its implementation Strongly agree 37 77,1% 21 80,8% 16 72,7% Rather agree 8 16,7% 3 11,5% 5 22,7% Rather disagree 2 4,2% 2 7,7% 0 0% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Increase the financial resources and significance of the Transversal Programme Strongly agree 5 10,4% 2 7,7% 3 13,6% Rather agree 17 35,4% 11 42,3% 6 27,3% Rather disagree 14 29,2% 8 30,8% 6 27,3% Strongly disagree 8 16,7% 3 11,5% 5 22,7% Do not know / cannot answer 4 8,3% 2 7,7% 2 9,1% 260

261 Total % % % 27. In your opinion, what is the best alternative for the integration of the Programme in the next programming period? All National Agencies National Authorities Frequency Frequency Frequency No change alternative: keeping the status quo of the current Programme (the centralised/decentralised subprogrammes 6 12,5% 4 15,4% 2 9,1% and actions as well as current management arrangements) Partial integration alternative: integrating some Programmespecific objectives, reducing the number of actions, further harmonisation in the Programme 24 50% 13 50% 11 50% documents and procedures, fostering co-operation between the Executive Agency and the National Agencies Full integration alternative: keeping only Programme-wide objectives, introducing common types of actions (mobility, partnerships, large projects and 18 37,5% 9 34,6% 9 40,9% networks, studies and other activities) across the subprogrammes, full standardisation of documents and procedures Do not know / Cannot answer 0 0% 0 0% 0 0% Total % % % 28. Please assess general success of the dissemination of the results of the following LLP sub-programmes in your country. All National Agencies National Authorities Frequency Frequency Frequency Comenius Very successful 15 31,2% 10 38,5% 5 22,7% Rather successful 23 47,9% 10 38,5% 13 59,1% Rather unsuccessful 4 8,3% 1 3,8% 3 13,6% Very unsuccessful 1 2,1% 1 3,8% 0 0% Do not know / can not answer 5 10,4% 4 15,4% 1 4,5% Erasmus Very successful 18 37,5% 11 42,3% 7 31,8% Rather successful 24 50% 12 46,2% 12 54,5% Rather unsuccessful 2 4,2% 0 0% 2 9,1% Very unsuccessful 0 0% 0 0% 0 0% Do not know / can not answer 4 8,3% 3 11,5% 1 4,5% Leonardo da Vinci 261

262 Very successful 12 25% 8 30,8% 4 18,2% Rather successful 27 56,2% 13 50% 14 63,6% Rather unsuccessful 3 6,2% 0 0% 3 13,6% Very unsuccessful 1 2,1% 1 3,8% 0 0% Do not know / can not answer 5 10,4% 4 15,4% 1 4,5% Grundtvig Very successful 9 18,8% 7 26,9% 2 9,1% Rather successful 23 47,9% 13 50% 10 45,5% Rather unsuccessful 9 18,8% 1 3,8% 8 36,4% Very unsuccessful 1 2,1% 1 3,8% 0 0% Do not know / can not answer 6 12,5% 4 15,4% 2 9,1% Total % % % 29. In your opinion, to what extent have good practices and results of LLP projects (except the individual mobility activities) been multiplied and exploited in your country as a result of the dissemination activities? All National Agencies National Authorities Frequency Frequency Frequency Within participating organisations: Good practices and results of the projects were adopted and applied by target groups of dissemination and exploitation activities within participating organisations Yes, frequently 26 54,2% 16 61,5% 10 45,5% Yes, sometimes 20 41,7% 10 38,5% 10 45,5% No, never 0 0% 0 0% 0 0% Do not know / cannot answer 2 4,2% 0 0% 2 9,1% Other organisations: Target groups of dissemination and exploitation activities outside participating organisations are informed / aware of the good practices and the results of the projects Yes, frequently 11 22,9% 10 38,5% 1 4,5% Yes, sometimes 33 68,8% 16 61,5% 17 77,3% No, never 1 2,1% 0 0% 1 4,5% Do not know / cannot answer 3 6,2% 0 0% 3 13,6% Other organisations: Target groups of dissemination and exploitation activities outside participating organisations adopted / applied the good practices and the results of the projects Yes, frequently 1 2,1% 1 3,8% 0 0% Yes, sometimes 39 81,2% 24 92,3% 15 68,2% No, never 2 4,2% 0 0% 2 9,1% Do not know / cannot answer 6 12,5% 1 3,8% 5 22,7% Decision makers at the local, regional, national and European level: Appropriate decision-makers at regional (or local), national or European levels (not including reporting to national agencies) are informed about the good practices and results of the projects Yes, frequently 14 29,2% 9 34,6% 5 22,7% Yes, sometimes 31 64,6% 16 61,5% 15 68,2% No, never 0 0% 0 0% 0 0% Do not know / cannot answer 3 6,2% 1 3,8% 2 9,1% Decision makers at the local, regional, national and European level: Good practices and results of the projects were adopted and mainstreamed into local, national or European policy on life-long learning Yes, frequently 3 6,2% 3 11,5% 0 0% Yes, sometimes 37 77,1% 20 76,9% 17 77,3% No, never 3 6,2% 2 7,7% 1 4,5% Do not know / cannot answer 5 10,4% 1 3,8% 4 18,2% 262

263 Total % % % 30. In your opinion, how could the effectiveness of the dissemination and exploitation activities be improved? All National Agencies National Authorities Frequency Frequency Frequency Shifting more resources and responsibilities for dissemination and exploitation of results to the beneficiaries Strongly agree 13 27,1% 7 26,9% 6 27,3% Rather agree 22 45,8% 14 53,8% 8 36,4% Rather disagree 11 22,9% 5 19,2% 6 27,3% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 2 4,2% 0 0% 2 9,1% Organising mandatory training events for dissemination and exploitation activities for the beneficiaries Strongly agree 4 8,3% 1 3,8% 3 13,6% Rather agree 29 60,4% 16 61,5% 13 59,1% Rather disagree 9 18,8% 7 26,9% 2 9,1% Strongly disagree 5 10,4% 1 3,8% 4 18,2% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% Increasing the centralised thematic co-ordination of dissemination and exploitation by the National Agencies / European Commission Strongly agree 9 18,8% 4 15,4% 5 22,7% Rather agree 22 45,8% 13 50% 9 40,9% Rather disagree 12 25% 6 23,1% 6 27,3% Strongly disagree 4 8,3% 2 7,7% 2 9,1% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% Increasing the co-operation between National Agencies, National Authorities, European Commission and Executive Agency in the dissemination and exploitation of results Strongly agree 17 35,4% 10 38,5% 7 31,8% Rather agree 25 52,1% 13 50% 12 54,5% Rather disagree 3 6,2% 2 7,7% 1 4,5% Strongly disagree 2 4,2% 0 0% 2 9,1% Do not know / cannot answer 1 2,1% 1 3,8% 0 0% Stimulating the use of project databases such as EVE and ADAM Strongly agree 16 33,3% 11 42,3% 5 22,7% Rather agree 22 45,8% 11 42,3% 11 50% Rather disagree 5 10,4% 3 11,5% 2 9,1% Strongly disagree 0 0% 0 0% 0 0% Do not know / cannot answer 5 10,4% 1 3,8% 4 18,2% Total % % % 31. Have the following mobility barriers had negative influences on the outputs and results of the LLP at the national level? (NATIONAL AGENCIES only) Frequency Language barriers 18 69,23% Differences in living standards between Participating Countries (e.g. not sufficient grants to cover expenses of individuals) 5 19,23% Heavy overload of educational and training staff of possible institutional beneficiaries 12 46,15% 263

264 Discouraging administrative burden of participation 12 46,15% Lack of recognition of skills and competences 15 57,69% Lack of control of the quality of mobility (e.g. certification) 0 0% Lack of awareness about mobility opportunities 2 7,69% Lack of awareness about added value of participation in mobility activities 4 15,38% Differences between national education and training systems 7 26,92% Total % 32. Please assess the contribution of the LLP s sectoral programmes to the EU priorities set out in the Education and Training Programme 2010? All National Agencies National Authorities Frequency Frequency Frequency Improving the quality and effectiveness of education and training systems in the EU (improving education and training for teachers and trainees, developing skills for the knowledge society, ensuring access to ICT for everyone, increasing recruitment to scientific and technical studies, making the best use of resources) Strong contribution 27 56,2% 15 57,7% 12 54,5% Weak contribution 20 41,7% 11 42,3% 9 40,9% No contribution 0 0% 0 0% 0 0% Do not know / cannot answer 1 2,1% 0 0% 1 4,5% Facilitating the access of all to education and training systems (Open learning environment, making learning more attractive, supporting active citizenship, equal opportunities and social cohesion) Strong contribution 22 45,8% 13 50% 9 40,9% Weak contribution 21 43,8% 11 42,3% 10 45,5% No contribution 3 6,2% 1 3,8% 2 9,1% Do not know / cannot answer 2 4,2% 1 3,8% 1 4,5% Opening up education and training systems to the wider world (strengthening the links with working life and research and society at large, developing the spirit of enterprise, improving foreign language learning, increasing mobility and exchange, strengthening the European co-operation) Strong contribution 35 72,9% 19 73,1% 16 72,7% Weak contribution 9 18,8% 5 19,2% 4 18,2% No contribution 2 4,2% 2 7,7% 0 0% Do not know / cannot answer 2 4,2% 0 0% 2 9,1% Total % % % 33. If you participated in peer learning activities and other fora of learning (networks, partnerships, clusters, experts groups, conferences and seminars or panels) organised by the European Commission or on its behalf in the framework of the LLP, please assess their usefulness to the following aspects. (NATIONAL AUTHORITIES only) Frequency Raising competences and awareness of national stakeholders in certain lifelong learning issues Very useful 8 36,4% Rather useful 7 31,8% Useful only to a limited extent 3 13,6% Not useful at all 0 0% Do not know / cannot answer 4 18,2% Facilitating consensus of main stakeholders in certain areas of education and training policy Very useful 2 9,1% Rather useful 7 31,8% 264

265 Useful only to a limited extent 8 36,4% Not useful at all 0 0% Do not know / cannot answer 5 22,7% Informing policy making at the EU or national level Very useful 9 40,9% Rather useful 6 27,3% Useful only to a limited extent 3 13,6% Not useful at all 0 0% Do not know / cannot answer 4 18,2% Monitoring progress towards the objectives and targets in lifelong learning Very useful 5 22,7% Rather useful 6 27,3% Useful only to a limited extent 5 22,7% Not useful at all 2 9,1% Do not know / cannot answer 4 18,2% Total % 34. To your opinion, how could contribution of the LLP to the EU education and training priorities and the European educational area be possibly improved? All National Agencies National Authorities Frequency Frequency Frequency Increase ownership and visibility of the EU priorities and the cooperation initiatives at the national 17 35,42% 11 42,31% 6 27,27% level Strengthen the involvement of national stakeholders in the cooperation 33 68,75% 20 76,92% 13 59,09% process at the EU level Improve the use of studies, comparative research and other evidence for informing policy 25 52,08% 12 46,15% 13 59,09% making and monitoring Increase the effectiveness of peer learning activities and other fora of 23 learning for informing policy making 47,92% 13 50% 10 45,45% and monitoring New structure of the Education and Training 2020 will provide more 12 25% 6 23,08% 6 27,27% flexibility for the LLP contribution Increased political ambition of education and training policy in the EU 2020 strategy will improve the 25 52,08% 12 46,15% 13 59,09% contribution of the LLP to the EU priorities Design of the future LLP should allow for a more flexible contribution to the EU priorities or 22 45,83% 11 42,31% 11 50% the co-operation initiatives Total % % % Respondents by country: 265

266 All National Agencies National Authorities Frequency Frequency Frequency EU ,2% 15 57,7% 11 50% EU ,4% 9 34,6% 8 36,4% Candidate and other Participating Countries 5 10,4% 2 7,7% 3 13,6% Total % % % 5.2. Survey of institutional beneficiaries 1.a Which action category(-ies) does your project(-s), implemented during the period , belong to? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Mobility , , , , , ,8 Partnerships , , , , , ,0 Other projects , , , , , ,1 Do not know / can not answer 417 5, ,1 19 1,8 31 1,7 97 6,3 13 5,1 Total , , , , , ,0 2. (Comenius) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Comenius Decntrl. Frequency COMENIUS Preparatory Visits ,0 COMENIUS Assistantships (ASSISTANTS) 160 5,5 COMENIUS Assistantships (HOST SCHOOLS) 119 4,1 COMENIUS Individual Pupil Mobility 63 2,2 COMENIUS In-Service Training for Teachers and other Educational Staff (IST) ,8 COMENIUS Multilateral school partnerships ,9 COMENIUS Bilateral school partnerships 175 6,0 COMENIUS Regio Partnerships 37 1,3 COMENIUS Multilateral projects 263 9,0 COMENIUS Multilateral networks 21,7 COMENIUS Accompanying measures 8,3 266

267 COMENIUS etwinning 144 4,9 Do not know/cannot answer 150 5,1 Total ,0 2. (Erasmus) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Erasmus Decentr. Frequency ERASMUS Preparatory Visits ,7 ERASMUS Organisation of Mobility ,5 ERASMUS Student Mobility for Studies ,6 ERASMUS Student Mobility for Placements ,6 ERASMUS Staff Mobility Teaching Assignments by HEI Teaching Staff and by Invited Staff from Enterprises ERASMUS Staff Mobility Training for HEI Staff at Enterprises and at HEI , ,5 ERASMUS Intensive Language Courses ,1 ERASMUS Intensive Programmes ,3 ERASMUS Academic Networks 49 4,7 ERASMUS Structural Networks 8,8 ERASMUS Curriculum Development Projects 51 4,8 ERASMUS Modernisation of Higher Education 14 1,3 ERASMUS Co-operation between Higher Education Institutions and Enterprises 39 3,7 ERASMUS Virtual Campuses 9,9 ERASMUS Accompanying Measures 19 1,8 Do not know/cannot answer 8,8 Total ,0 2. (Leonardo da Vinci) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Leonardo Decentr. Frequency LEONARDO DA VINCI Preparatory Visits ,4 LEONARDO DA VINCI IVT (Initial Vocational Training) ,3 LEONARDO DA VINCI PLM (People in the Labour Market) ,6 LEONARDO DA VINCI VETPRO (VET Professionals) ,9 267

268 LEONARDO DA VINCI Partnerships ,6 LEONARDO DA VINCI Transfer of innovation ,6 LEONARDO DA VINCI Development of innovation 47 2,5 LEONARDO DA VINCI Multilateral networks 27 1,5 LEONARDO DA VINCI Accompanying measures 5,3 Do not know/cannot answer 37 2,0 Total ,0 2. (Grundtvig) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Grundtvig Decentr. Frequency GRUNDTVIG Preparatory Visits ,2 GRUNDTVIG Visits and Exchanges for adult education staff ,9 GRUNDTVIG Assistantships 18 1,2 GRUNDTVIG In-Service training for adult education staff ,3 GRUNDTVIG Workshops 145 9,5 GRUNDTVIG Learning Partnerships ,8 GRUNDTVIG Senior Volunteering Projects 24 1,6 GRUNDTVIG Multilateral project 106 6,9 GRUNDTVIG Multilateral networks 22 1,4 GRUNDTVIG Accompanying measures 8,5 Do not know/cannot answer 75 4,9 Total ,0 2. (JnM) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Jean Monnet Frequency JEAN MONNET Chairs 15 5,8 JEAN MONNET Chairs Ad Personam 9 3,5 JEAN MONNET Centres of Excellence 10 3,9 JEAN MONNET Teaching Modules 15 5,8 JEAN MONNET Information and research activities 14 5,4 JEAN MONNET Associations of Professors and Researchers specialising in European Integration 4 1,6 268

269 JEAN MONNET Multilateral Research groups 3 1,2 JEAN MONNET European associations active in education and training 0 0 Do not know/cannot answer (Transversal) Which action (-s) was your project (-s) / activity (-ies) funded under during the period ? Transversal Frequency KA1 Study Visits for Education and Vocational Training Specialists 5 1,9 KA1 Studies and Comparative Research 7 2,7 KA2 New materials / online courses / awareness raising 12 4,7 KA2 Multilateral networks 11 4,3 KA2 Accompanying measures 3 1,2 KA3 Multilateral projects 17 6,6 KA3 Multilateral networks 4 1,6 KA4 Multilateral projects 9 3,5 Do not know/cannot answer 7 2,7 3. Your organisation is: Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Education provider (including higher education institutions) Association (including students associations, employers, employees, etc.) , , , , , , ,1 23,8 5, , ,4 17 6,6 Enterprise 216 2,7 4, ,4 76 5,0 12 4,7 Public authority 400 5, ,8 14 1,3 99 5,3 94 6,1 12 4,7 Public service provider 159 2,0 49 1,7 3,3 36 1,9 59 3,9 6 2,3 Counselling body 41,5 3, ,0 16 1,0 0 0 Research centre 64,8 2,1 1,1 30 1,6 18 1,2 10 3,9 269

270 Non-governmental organisation 458 5,8 8,3 6, , ,5 16 6,2 Other (Please specify) 447 5, ,6 3, , ,8 19 7,4 Total , , , , , ,0 4a. In which country is your organisation located? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Austria 288 3, ,5 35 3,3 75 4,1 64 4,2 5 1,9 Belgium 245 3,1 62 2,1 31 2,9 66 3,6 60 3,9 21 8,2 Bulgaria 68,9 28 1, ,1 10,7 4 1,6 Cyprus 58,7 13,4 4,4 16,9 15 1,0 6 2,3 Czech Republic 235 3,0 81 2,8 33 3,1 62 3,3 49 3,2 1,4 Denmark 168 2,1 55 1,9 26 2,5 40 2,2 35 2,3 3 1,2 Estonia 104 1,3 30 1,0 16 1,5 31 1,7 23 1,5 1,4 Finland 221 2,8 67 2,3 33 3,1 53 2,9 59 3,9 1,4 France , , , , ,2 6 2,3 Germany , , , , ,9 16 6,2 Greece 31,4 6,2 6,6 6,3 4,3 8 3,1 Hungary 175 2,2 57 2,0 25 2,4 57 3,1 27 1,8 2,8 Iceland 27,3 14,5 1,1 9,5 3,2 0 0 Ireland 94 1,2 40 1,4 2,2 10,5 35 2,3 5 1,9 Italy 597 7, ,5 57 5,4 69 3, , ,3 Latvia 101 1,3 41 1, ,8 22 1,4 1,4 Liechtenstein 2,0 1, , Lithuania 262 3, ,5 34 3,2 36 1,9 58 3,8 1,4 Luxembourg 24,3 5, ,5 8,5 2,8 Malta 16,2 6,2 2,2 4,2 3,2 0 0 the Netherlands 104 1,3 35 1,2 14 1,3 15,8 30 2,0 7 2,7 Norway 93 1,2 32 1,1 9,9 32 1,7 14,9 1,4 Poland 390 5, ,5 57 5,4 90 4,9 69 4,5 3 1,2 Portugal 194 2,5 73 2,5 37 3,5 31 1,7 43 2,8 5 1,9 Romania 246 3, ,4 23 2,2 60 3,2 46 3,0 6 2,3 Slovakia 60,8 14,5 11 1,0 21 1,1 11,7 2,8 Slovenia 66,8 22,8 18 1,7 15,8 8,5 2,8 Spain , , , , ,0 18 7,0 270

271 Sweden 140 1,8 59 2,0 10 1,0 34 1,8 34 2,2 2,8 Turkey 186 2,4 62 2,1 28 2,7 26 1,4 55 3,6 2,8 United Kingdom 658 8, ,3 57 5, , ,1 8 3,1 Other (please specify) 32,4 7,2 1,1 2,1 5,3 16 6,2 Total , , , , , ,0 4b. What is an approximate size of your organisation in terms of employed staff? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Up to and more Total Was (were) your project(-s)/activities, funded by the programme, successful (or likely to be successful) in reaching the following target groups of your project(-s)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Learners (pupils, students, trainees, adult learners) very successful , , , , , ,0 rather successful , , , , , ,3 rather unsuccessful 121 1,5 36 1,2 16 1,5 21 1,1 34 2,2 3 1,2 very unsuccessful 47,6 16,5 7,7 14,8 6,4 1,4 271

272 do not know / cannot answer/ is not applicable 555 7, ,9 25 2, , ,4 19 7,4 Teachers, trainers very successful , , , , , ,4 rather successful , , , , , ,0 rather unsuccessful 180 2,3 58 2,0 60 5,7 22 1,2 27 1,8 3 1,2 very unsuccessful 60,8 21,7 12 1,1 12,6 11,7 1,4 do not know / cannot answer/ is not applicable 503 6,4 46 1,6 68 6, , ,6 16 6,2 Other staff of teaching institutions very successful , , , , , ,0 rather successful , , , , , ,6 rather unsuccessful 362 4, ,5 79 7,5 48 2,6 51 3,3 7 2,7 very unsuccessful 74,9 29 1,0 16 1,5 12,6 8,5 4 1,6 do not know / cannot answer/ is not applicable , , , , , ,0 People in the labour market very successful 623 7,9 75 2,6 42 4, , ,0 25 9,7 rather successful , ,7 74 7, , , ,1 rather unsuccessful 233 3,0 83 2,8 25 2,4 40 2,2 54 3,5 12 4,7 very unsuccessful 84 1,1 40 1,4 12 1,1 15,8 13,8 0 0 do not know / cannot answer/ is not applicable , , , , , ,1 People outside the labour market (the retired or unemployed) very successful 349 4,4 36 1,2 11 1,0 97 5, ,7 3 1,2 rather successful 630 8, ,9 20 1, , , ,8 rather unsuccessful 219 2,8 97 3,3 10 1,0 35 1,9 46 3,0 8 3,1 very unsuccessful 123 1,6 61 2,1 13 1,2 18 1,0 18 1,2 6 2,3 do not know / cannot answer/ is not applicable , , , , , ,9 People excluded from the education and training systems (early school leavers) very successful 325 4,1 73 2,5 9,9 92 5, ,7 8 3,1 rather successful 570 7, ,5 19 1, , ,5 14 5,4 rather unsuccessful 219 2,8 92 3,2 10 1,0 30 1,6 56 3,7 7 2,7 272

273 very unsuccessful 127 1,6 59 2,0 12 1,1 20 1,1 21 1,4 4 1,6 do not know / cannot answer/ is not applicable , , , , , ,7 People in rural and remote communities very successful 599 7, ,9 23 2, , ,8 16 6,2 rather successful , ,8 33 3, , ,6 21 8,2 rather unsuccessful 196 2,5 78 2,7 9,9 41 2,2 52 3,4 2,8 very unsuccessful 111 1,4 45 1,5 12 1,1 18 1,0 19 1,2 4 1,6 do not know / cannot answer/ is not applicable , , , , , ,2 Linguistic minorities very successful 416 5, ,2 24 2,3 68 3, ,6 11 4,3 rather successful 688 8, ,2 41 3,9 87 4, ,4 19 7,4 rather unsuccessful 218 2,8 84 2,9 18 1,7 38 2,1 54 3,5 4 1,6 very unsuccessful 118 1,5 46 1,6 7,7 21 1,1 25 1,6 6 2,3 do not know / cannot answer/ is not applicable , , , , , ,1 Migrants very successful 506 6, ,6 17 1,6 80 4, ,4 14 5,4 rather successful , ,8 52 4, , ,3 19 7,4 rather unsuccessful 218 2,8 86 3,0 14 1,3 36 1,9 52 3,4 7 2,7 very unsuccessful 121 1,5 48 1,6 13 1,2 19 1,0 22 1,4 6 2,3 do not know / cannot answer/ is not applicable , , , , , ,6 People with disabilities, special needs and socially disadvantaged very successful 779 9, ,2 38 3, , ,4 10 3,9 rather successful , , , , , ,3 rather unsuccessful 201 2,6 69 2,4 27 2,6 35 1,9 46 3,0 6 2,3 very unsuccessful 132 1,7 53 1,8 19 1,8 15,8 29 1,9 6 2,3 do not know / cannot answer/ is not applicable , , , , , ,9 Education providers (including higher education institutions) very successful , , , , , ,0 rather successful , , , , , ,2 273

274 rather unsuccessful 181 2,3 62 2,1 19 1,8 41 2,2 42 2,7 4 1,6 very unsuccessful 66,8 23,8 4,4 16,9 15 1,0 1,4 do not know / cannot answer/ is not applicable , , , , ,7 21 8,2 Higher education institutions in third countries very successful 334 4, ,5 56 5,3 63 3,4 70 4, ,1 rather successful 460 5, ,6 98 9,3 95 5,1 90 5, ,3 rather unsuccessful 197 2,5 60 2,1 25 2,4 37 2,0 46 3,0 7 2,7 very unsuccessful 119 1,5 41 1,4 16 1,5 19 1,0 28 1,8 6 2,3 do not know / cannot answer/ is not applicable , , , , , ,6 Associations (including associations of students, employers, employees, etc. very successful 640 8, ,7 50 4, , , ,7 rather successful , , , , , ,0 rather unsuccessful 215 2,7 70 2,4 24 2,3 47 2,5 46 3,0 7 2,7 very unsuccessful 100 1,3 50 1,7 7,7 17,9 16 1,0 4 1,6 do not know / cannot answer/ is not applicable , , , , , ,8 Enterprises, including SMEs very successful 547 7,0 70 2,4 83 7, ,6 77 5, ,0 rather successful , , , , , ,5 rather unsuccessful 257 3,3 82 2,8 36 3,4 56 3,0 57 3,7 8 3,1 very unsuccessful 107 1,4 53 1,8 14 1,3 15,8 14,9 1,4 do not know / cannot answer/ is not applicable , , , , , ,4 Public (national, regional, local) authorities very successful , ,8 59 5, , , ,2 rather successful , , , , , ,5 rather unsuccessful 263 3,3 96 3,3 27 2,6 53 2,9 60 3,9 9 3,5 very unsuccessful 87 1,1 42 1,4 8,8 16,9 10,7 2,8 do not know / cannot answer/ is not applicable , , , , , ,0 Public service providers very successful 510 6, ,9 38 3, , ,4 25 9,7 274

275 rather successful , ,4 54 5, , , ,1 rather unsuccessful 238 3,0 86 3,0 20 1,9 54 2,9 58 3,8 7 2,7 very unsuccessful 90 1,1 43 1,5 12 1,1 16,9 11,7 1,4 do not know / cannot answer/ is not applicable , , , , , ,0 Counselling bodies very successful 341 4, ,5 19 1,8 90 4,9 89 5,8 17 6,6 rather successful 646 8, ,2 39 3, , , ,6 rather unsuccessful 223 2,8 84 2,9 15 1,4 44 2,4 54 3,5 7 2,7 very unsuccessful 104 1,3 46 1,6 13 1,2 18 1,0 20 1,3 2,8 do not know / cannot answer/ is not applicable , , , , , ,6 Research centres very successful 350 4,5 75 2,6 58 5,5 92 5,0 71 4, ,5 rather successful 640 8, , , , , ,7 rather unsuccessful 211 2,7 74 2,5 23 2,2 38 2,1 58 3,8 6 2,3 very unsuccessful 114 1,5 48 1,6 11 1,0 17,9 27 1,8 2,8 do not know / cannot answer/ is not applicable , , , , , ,0 Non-governmental organisations very successful 566 7, ,5 28 2, , , ,3 rather successful 760 9, ,7 52 4, , , ,8 rather unsuccessful 197 2,5 72 2,5 21 2,0 36 1,9 43 2,8 10 3,9 very unsuccessful 107 1,4 52 1,8 8,8 15,8 18 1,2 1,4 do not know / cannot answer/ is not applicable , , , , , ,6 Other (please specify) very successful 84 1,1 18,6 2,2 22 1,2 31 2,0 7 2,7 rather successful 62,8 18,6 3,3 11,6 27 1,8 2,8 rather unsuccessful 19,2 8,3 1,1 3,2 5,3 1,4 very unsuccessful 21,3 13,4 1,1 4,2 2,1 0 0 do not know / cannot answer/ is not applicable , , , , , ,6 Total , , , , , ,0 275

276 6. What are the main activities and outputs of your project(-s)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Exchange of information, experience and good practice , , , , , ,6 Preparatory visits , , , , ,3 25 9,7 Mobility/virtual mobility/transnational meeting Framework for the organisation of mobility activities Partnership/network/E uropean association , , , , , , , , , , , , , , , , , ,9 Linguistic preparation , , , , , ,3 Preparation / delivery of a new curriculum/module/pr ogramme/course/ or learning/training methodology New teaching/training material New teaching/training methodology or (educational) strategy New guidance/counselling tool/method Comparative study/case study/research report(s)/article(s)/oth er publications, recommendations , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,4 276

277 Seminars, workshops, working groups, conferences, summer universities, exhibitions / fairs Information/promotio nal campaign or other awareness-raising activity/tool Tools and methods for the recognition and evaluation of competences and skills Activities in the area of quality assurance Creation of a technical object, drawing, arts object, performance (e.g. theatre play, musical, etc.) Dissemination and exchange of project results between institutions Implementation and mainstreaming of project results in educational policy on local, national or European level New commercial products New indicators/tools/datab ases of good practice Virtual learning spaces (e.g. websites etc.) New application of informationcommunication technologies , , , , , , , , , , , , , , , , , , , , , , , , , ,1 57 5,4 86 4, ,9 7 2, , , , , , , , ,3 90 8, , , , ,1 47 1,6 14 1,3 55 3,0 34 2,2 12 4, , ,1 83 7, , , , , , , , , , , ,3 97 9, , , ,8 277

278 Creation of new products for informationcommunication technologies (software etc) Promoting skills training for informationcommunication technologies Strategies, action plans, other documents 382 4, ,6 34 3,2 83 4,5 48 3,1 21 8, , ,5 70 6, , ,0 24 9, , , , , , ,9 New/pilot projects , ,8 94 8, , , ,5 Work places 705 9, , , ,5 67 4,4 7 2,7 Other (please specify) 186 2,4 44 1,5 21 2,0 59 3,2 47 3,1 13 5,1 Total , , , , , ,0 7. Have the following external factors influenced the results of your project(-s)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Mobility barriers (e.g. different living standards between countries, insurance policies, legal obstacles, etc. ) Strong influence 451 5, ,4 89 8,5 99 5,3 92 6,0 5 1,9 Some influence , , , , , ,1 No influence , , , , , ,9 Do not know / cannot answer 505 6, ,4 52 4,9 90 4, ,9 19 7,4 Language barriers Strong influence 618 7, , , , ,4 6 2,3 Some influence , , , , , ,0 No influence , , , , , ,5 Do not know / cannot answer 219 2,8 79 2,7 19 1,8 38 2,1 60 3,9 8 3,1 Changed financial position of the organisations/individuals participating in the project (e.g. owing to economic downturn) Strong influence 419 5, , , ,1 65 4,2 6 2,3 Some influence , , , , , ,8 278

279 No influence , , , , , ,7 Do not know / cannot answer , , , , , ,9 Lack of support from potential socio-economic partners, able to participate in the development and implementation of the project Strong influence 254 3,2 86 3,0 44 4,2 64 3,5 45 2,9 5 1,9 Some influence , , , , , ,7 No influence , , , , , ,5 Do not know / cannot answer , , , , , ,2 Lack of capacity/willingness of potential individual beneficiaries (participants of the programme) to participate in the project Strong influence 282 3,6 74 2,5 65 6,2 71 3,8 56 3,7 4 1,6 Some influence , , , , , ,2 No influence , , , , , ,6 Do not know / cannot answer , , , , ,4 23 8,9 Unfavourable personal values and attitudes of potential individual beneficiaries (participants of the programme Strong influence 188 2,4 75 2,6 25 2,4 37 2,0 42 2,7 2,8 Some influence , , , , ,1 23 8,9 No influence , , , , , ,6 Do not know / cannot answer , , , , , ,9 Changes in national educational policy Strong influence 253 3,2 95 3,3 50 4,8 48 2,6 41 2,7 5 1,9 Some influence , , , , , ,7 No influence , , , , , ,6 Do not know / cannot answer , , , , ,8 23 8,9 Other (please specify Strong influence 86 1,1 32 1,1 13 1,2 22 1,2 14,9 4 1,6 Some influence 53,7 19,7 6,6 13,7 11,7 3 1,2 No influence 467 5, ,4 24 2, , ,9 14 5,4 Do not know / cannot answer , , , , ,7 17 6,6 Total , , , , , ,0 8. What influence did your participation in the <sub-programme> programme have on your organisation? 279

280 Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Professional competences of staff and trainees in your organisation (improved specific skills or enhanced specific knowledge Strong influence , ,6 89 8, , , ,2 Some influence , , , , , ,4 No influence 337 4, , ,7 80 4,3 77 5,0 5 1,9 Do not know / cannot answer 249 3,2 84 2,9 52 4,9 61 3,3 57 3,7 4 1,6 Specific (professional or academic) competences of learners (students, adult learners, pupils etc.) in your organisation Strong influence , , , , , ,9 Some influence , , , , , ,9 No influence 461 5, ,5 36 3, , ,3 18 7,0 Do not know / cannot answer 471 6, ,7 28 2, , ,8 18 7,0 Key competences of staff and trainees in your organisation (such as learning to learn, social and civic competences, sense of initiative and entrepreneurship, cultural awareness and expression, mathematical competence and basic competences in science and technology, digital competence etc. Strong influence , , , , , ,6 Some influence , , , , , ,7 No influence 688 8, , , , ,8 10 3,9 Do not know / cannot answer 629 8, , , , ,8 14 5,4 Key competences of learners (students, adult learners, pupils etc.) in your organisation Strong influence , , , , , ,3 Some influence , , , , , ,1 No influence 636 8, ,3 56 5, , ,0 23 8,9 Do not know / cannot answer 674 8, ,3 61 5, , , ,1 Foreign language skills (listening, speaking, writing) of staff and trainees Strong influence , , , , , ,7 Some influence , , , , , ,3 No influence , , , , , ,4 Do not know / cannot answer 478 6, ,8 63 6, , ,8 18 7,0 Foreign language skills (listening, speaking, writing) of learners Strong influence , , , , , ,7 Some influence , , , , , ,1 280

281 No influence , ,7 49 4, , , ,8 Do not know / cannot answer 537 6, ,5 38 3, , , ,7 Improvement of content/curriculum of education/training Strong influence , , , , , ,2 Some influence , , , , , ,6 No influence , , , , ,5 18 7,0 Do not know / cannot answer 700 8, , , , ,7 13 5,1 Improvement of practice and techniques of education/training Strong influence , , , , , ,8 Some influence , , , , , ,3 No influence , , , , ,6 15 5,8 Do not know / cannot answer 696 8, , , , ,4 13 5,1 Improvement in application of information-communication technologies Strong influence , , , , , ,7 Some influence , , , , , ,3 No influence , , , , , ,1 Do not know / cannot answer , , , , ,3 19 7,4 Accessibility to education/training services for different target groups Strong influence , ,6 92 8, , , ,5 Some influence , , , , , ,8 No influence , , , , , ,5 Do not know / cannot answer , , , , , ,7 Better organisation of mobility, improvement of procedures for supporting mobile learners (students, adult learners, pupils etc.) in your organisation Strong influence , , , , ,3 24 9,3 Some influence , , , , , ,8 No influence , ,5 75 7, , , ,2 Do not know / cannot answer , ,5 55 5, , , ,0 Development / mainstreaming of ICT-based content and practice in your organisation Strong influence , ,0 82 7, , , ,3 Some influence , , , , , ,1 No influence , , , , , ,1 281

282 Do not know / cannot answer , , , , , ,2 Total , , , , , ,0 9. (Comenius) What influence did your participation in the Comenius programme have on the following issues in your specific thematic area? Frequency Improving the learning environment of the pupils Strong influence Some influence No influence Do not know / cannot answer Strengthening the quality of in-service training and professional development for serving teachers Strong influence Some influence No influence Do not know / cannot answer Strengthening the quality of initial training for intending and initial teachers Strong influence Some influence No influence Do not know / cannot answer Strengthening of educational partnerships and networks Strong influence Some influence No influence Do not know / cannot answer Improved access to information about European and wider global educational partnership opportunities Strong influence Some influence No influence Do not know / cannot answer Development of a European and/or wider global dimension to pupils learning Strong influence Some influence No influence Do not know / cannot answer Sharing of educational good practice on a European level Strong influence Some influence No influence Do not know / cannot answer Improved European links and interaction between Local Authorities, schools, school education organisations, business Strong influence

283 Some influence No influence Do not know / cannot answer Other important influences (please specify Strong influence Some influence No influence Do not know / cannot answer Total (Erasmus) What influence did your participation in the Erasmus programme have on the following issues in your specific thematic area? Frequency Creating links between higher education institutions and enterprises, private sector Strong influence Some influence No influence Do not know / cannot answer Improvement of the international profile of the organisation Strong influence Some influence No influence Do not know / cannot answer Wider experience and knowledge of the staff in education systems at international level or in the host countries of the mobility Strong influence Some influence No influence Do not know / cannot answer Establishing or enlarging the network of partners or contacts, raising participation of staff in professional networks Strong influence Some influence No influence Do not know / cannot answer Enhanced transnational cooperation in the development of teaching content and practice Strong influence Some influence No influence Do not know / cannot answer Greater awareness of the European dimension in higher education Strong influence Some influence No influence Do not know / cannot answer Improvement of the performance of higher education institutions through specific measures/actions or strategies created 283

284 Strong influence Some influence No influence Do not know / cannot answer Fostering practices of quality assurance Strong influence Some influence No influence Do not know / cannot answer Other important influences (please specify) Strong influence Some influence 5.5 No influence Do not know / cannot answer Total (Leonardo da Vinci) What influence did your participation in the Leonardo da Vinci programme have on the following issues in your specific thematic area? Frequency Raising competences of teachers / trainers on new demands from society (new target groups, topics, learning style) Strong influence Some influence No influence Do not know / cannot answer Raising competences of early school leavers Strong influence Some influence No influence Do not know / cannot answer Raising competences of people in the labour market with low qualification levels Strong influence Some influence No influence Do not know / cannot answer Better awareness of the pedagogical approach in the host country of the mobility programme Strong influence Some influence No influence Do not know / cannot answer Better awareness of VET system in the host country of the mobility programme Strong influence Some influence No influence Do not know / cannot answer Changes of attitudes and perceptions regarding European integration Strong influence Some influence

285 No influence Do not know / cannot answer Greater awareness of the European dimension of VET Strong influence Some influence No influence Do not know / cannot answer New contacts within the VET field Strong influence Some influence No influence Do not know / cannot answer Providing internships for students Strong influence Some influence No influence Do not know / cannot answer Encouraging the participation of social partner organisations and small and medium-sized enterprises in networks Strong influence Some influence No influence Do not know / cannot answer Other important influences (please specify) Strong influence Some influence No influence Do not know / cannot answer Total (Grundtvig) What influence did your participation in the Grundtvig programme have on the following issues in your specific thematic area? Frequency Reaching hard to reach target groups which do not generally tend to take part in educational activities Strong influence Some influence No influence Do not know / cannot answer Raising competences of people lacking basic education and qualification Strong influence Some influence No influence Do not know / cannot answer Raising competences of people living in rural or disadvantaged areas Strong influence Some influence No influence Do not know / cannot answer

286 Raising competences of people who are disadvantaged for socio economic reasons Strong influence Some influence No influence Do not know / cannot answer Raising competences of teachers / trainers on new demands from society (e.g. new target groups, topics, learning styles Strong influence Some influence No influence Do not know / cannot answer Raising competences of managers on the new demands of society (e.g. new target groups, topics, learning styles, methods Strong influence Some influence No influence Do not know / cannot answer Changes of attitudes and perceptions regarding the host country of mobility programme Strong influence Some influence No influence Do not know / cannot answer Awareness of pedagogical approach in host country of mobility programme Strong influence Some influence No influence Do not know / cannot answer Better awareness of adult education system in host country of mobility programme Strong influence Some influence No influence Do not know / cannot answer Changes of attitudes and perceptions regarding European integration Strong influence Some influence No influence Do not know / cannot answer Greater awareness of the European dimension in adult education Strong influence Some influence No influence Do not know / cannot answer Greater awareness of other actors in the field of adult education in general Strong influence Some influence No influence Do not know / cannot answer New contacts within the field of adult education Strong influence Some influence

287 No influence Do not know / cannot answer Other important influences (please specify) Strong influence Some influence No influence Do not know / cannot answer Total (Jean Monnet) What influence did your participation in the Jean Monnet programme have on the following issues in your specific thematic area? Frequency Stimulating excellence in teaching, research and reflection in European integration studies in higher education institutions within EU Strong influence Some influence No influence Do not know / cannot answer 2.8 Stimulating excellence in teaching, research and reflection in European integration studies in higher education institutions outside EU Strong influence Some influence No influence Do not know / cannot answer 2.8 Exploiting multidisciplinary resources in academic activities Strong influence Some influence No influence 1.4 Do not know / cannot answer 2.8 Achieving a transnational dimension in academic activities Strong influence Some influence No influence 2.8 Do not know / cannot answer 1.4 Enhancing knowledge and awareness generally among specialists academics of issues relating to European integration Strong influence Some influence No influence 1.4 Do not know / cannot answer 2.8 Addressing students at undergraduate, graduate and post-graduate levels Strong influence Some influence No influence 0 0 Do not know / cannot answer 1.4 Addressing students who do not automatically come into contact with European integration studies Strong influence

288 Some influence No influence Do not know / cannot answer 2.8 Enhancing knowledge and awareness among European citizens generally of issues relating to European integration Strong influence Some influence No influence Do not know / cannot answer Addressing specific civil society groups (e.g. adults, primary and secondary school teachers) Strong influence 2.8 Some influence No influence Do not know / cannot answer Other important influences (please specify) Strong influence 2.8 Some influence 2.8 No influence 1.4 Do not know / cannot answer Total (Transversal) What influence did your participation in the Transversal programme have on the following issues in your specific thematic area? Frequency Raising the competences of policy makers and education/training specialists on the new demands of society Strong influence Some influence No influence 1.4 Do not know / cannot answer Informing policy-making Strong influence Some influence No influence Do not know / cannot answer Raising the competence of people with no experience of language learning Strong influence Some influence No influence Do not know / cannot answer Contribution to the preservation of regional/minority languages and encouragement of learning less widely used languages Strong influence Some influence No influence Do not know / cannot answer Increasing knowledge of information-communication technologies related teaching methods/content and their exploitation 288

289 Strong influence Some influence No influence Do not know / cannot answer 2.8 Increasing the application of information-communication technologies in educational practice Strong influence Some influence No influence Do not know / cannot answer 1.4 Preparing/ fostering an environment for the active use of information-communication technologies in educational practice Strong influence Some influence No influence Do not know / cannot answer 1.4 Other important influences (please specify) Strong influence 2.8 Some influence 0 0 No influence 2.8 Do not know / cannot answer Total (Centralised) Did your project(-s) involve participation in multilateral networks? Frequency Yes No Do not know / cannot answer Total (Centralised) What influence did your participation in the network have on the following? Promoting innovation and best practice Frequency Strong influence Some influence No influence 2.8 Do not know / cannot answer 2.8 Promoting European cooperation Strong influence Some influence No influence 1.4 Do not know / cannot answer Building partnerships of specific target groups Strong influence Some influence No influence 2.8 Do not know / cannot answer

290 Assisting in networking between individuals and institutions benefiting from the Programme Strong influence Some influence No influence Do not know / cannot answer Contributing to research, other empirical evidence Strong influence Some influence No influence Do not know / cannot answer Informing policy making Strong influence Some influence No influence Do not know / cannot answer Raising awareness about a specific thematic issue among the target groups Strong influence Some influence No influence 2.8 Do not know / cannot answer Assisted in embedding information-communication technologies in longer term policy and practice Strong influence Some influence No influence Do not know / cannot answer Other important influences (please specify Strong influence Some influence 2.8 No influence 1.4 Do not know / cannot answer Total (Centralised) The network you are (or previously were) participating in constitute (several answers possible): Frequency A forum A platform Do not know / cannot answer Total Would your project(-s)/activities have taken place without funding from the <sub-programme>? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t 290

291 Most likely not Most likely yes, with the help of other national/international Most likely yes, with the help of other national/international Most likely yes, using own funds Most likely yes, using own funds, but with a lower intensity Do not know / cannot answer Total Will the results of your project(-s) be sustainable and accessible after the end of the project? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen cy Freque ncy Frequen cy Freque ncy Percen t Freque ncy Freque ncy Percen Yes, very likely Yes, rather likely No, rather unlikely No, very unlikely Do not know / can not answer Total t 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin

292 very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Disability / special needs very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Sexual orientation very successful rather successful rather unsuccessful very unsucessful issue was not relevant / project did not address this issue

293 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer

294 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue Age 294

295 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue very successful rather successful

296 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue rather unsuccessful

297 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue very unsucessful

298 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue issue was not relevant / project did not address this issue

299 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer

300 15. Was your project(-s) successful in addressing the following equal opportunity issues? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Gender very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Racial / ethnic origin very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Roma very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue do not know / cannot answer Religion / belief very successful rather successful rather unsuccessful very unsuccessful issue was not relevant / project did not address this issue Total

301 16. Did you face the following problems and obstacles while trying to address issues of equal opportunities in your project(-s)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t It is difficult to reach the target groups to whom the issues of equal opportunities would be relevant Yes Yes, to some extent No Do not know / cannot answer Insufficient information available as to how the issues of equal opportunities could be addressed in the project applications Yes Yes, to some extent No Do not know / cannot answer The issues of equal opportunities are not prioritised enough in the Calls for Proposals of the LLP Yes Yes, to some extent No Do not know / cannot answer Addressing the issues of equal opportunities requires additional financial resources, human resources and time Yes Yes, to some extent No Do not know / cannot answer There are difficulties in involving educational and other institutions in socio-economically disadvantaged areas Yes Yes, to some extent No Do not know / cannot answer The programme management is not flexible enough to address the issues of equal opportunities Yes Yes, to some extent No Do not know / cannot answer Other important problems (please specify) Yes Yes, to some extent No Do not know / cannot answer

302 Total How likely is it that your organisation would have sought to address the issue of equal opportunities in ensuring access to lifelong learning, if your project(-s) was not supported by the LLP? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen cy Freque ncy Frequen cy Freque ncy Percen t Freque ncy Freque ncy Percen Very likely Somewhat likely Rather unlikely Unlikely Do not know / cannot answer Total t 18. How would you assess your overall satisfaction with participation in the <sub-programme> programme? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen cy Freque ncy Frequen cy Freque ncy Percen t Freque ncy Freque ncy Percen Very satisfied Rather satisfied Rather unsatisfied Very unsatisfied Do not know / cannot answer Total t 19. How would you evaluate the following aspects of the implementation of the Programme? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Application timing is well coordinated with the relevant education processes Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Requirements for applicants are clear, the application procedure is user friendly Strongly agree Rather agree

303 Rather disagree Strongly disagree Do not know / cannot answer Assessment criteria and their application are clear and transparent Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Selection and contracting of successful applications was timely and took the appropriate time Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Expenditure eligibility requirements are clear Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Procedures pertaining to financial management of the project (payment arrangements, requirements for accounting and justification of expenditure, etc.) are clear Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Payment processing procedures were timely and took the appropriate time Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer The financing conditions (such as 10 per cent threshold applicable to budget re-allocations, percentage of indirect costs and percentage of first instalment) allow for Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer

304 Duration of the project and timing of reporting are well coordinated with the relevant education processes (e.g. duration and timing of study programmes) Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Other practicalities (please specify) Strongly agree Rather agree Rather disagree Strongly disagree Do not know / cannot answer Total How would you assess the support and guidance you received from the National Agency (or Education, Audiovisual and Culture Executive Agency (EACEA) for centralised actions) during the following phases of your participation in the Programme? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Information on funding opportunities very positive Rather positive Rather negative Very negative Do not know / cannot answer / not applicable Application requirements and procedures very positive Rather positive Rather negative Very negative Do not know / cannot answer / not applicable Financial and contractual management, reporting very positive Rather positive Rather negative Very negative Do not know / cannot answer / not applicable Thematic monitoring very positive Rather positive

305 Rather negative Very negative Do not know / cannot answer / not applicable Dissemination of the project results very positive Rather positive Rather negative Very negative Do not know / cannot answer / not applicable Total To what extent did the priorities of your project match the priorities of your organisation? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Very strong match Somewhat matched The match was limited Did not match Do not know / cannot answer Total How important are the following issues to your organisation? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Development of quality, performance, and innovation in providing lifelong learning activities Very important Rather important Rather unimportant Not important Do not know / cannot answer Promoting active participation of learners in the learning process - and understanding of the learning process Very important Rather important Rather unimportant Not important Do not know / cannot answer Development of a European dimension to learning (increasing awareness of the EU, building cross-national partnerships, transferring standards and best-practices, etc.) 305

306 Very important Rather important Rather unimportant Not important Do not know / cannot answer Improving the attractiveness and accessibility of learning opportunities Very important Rather important Rather unimportant Not important Do not know / cannot answer Promoting language learning and linguistic diversity Very important Rather important Rather unimportant Not important Do not know / cannot answer Development of innovative ICT-based content, services, pedagogies and practice Very important Rather important Rather unimportant Not important Do not know / cannot answer Improving (local, national, regional) public policy on Life-long learning Very important Rather important Rather unimportant Not important Do not know / cannot answer Total How successful has the implementation of the dissemination/exploitation plan of your project(-s) been (likely to be in future)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Very successful Rather successful Rather unsuccessful Very unsuccessful Do not know / cannot answer Total

307 24. Has your organisation applied the following instruments in order to disseminate the results and experiences gained from your participation in the <sub-programme> programme? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Internet websites Yes Yes, to some extent No Do not know / cannot answer Manuals, readers, other material Yes Yes, to some extent No Do not know / cannot answer Handbooks, books Yes Yes, to some extent No Do not know / cannot answer Exhibitions Yes Yes, to some extent No Do not know / cannot answer Studies, reports, other publications Yes Yes, to some extent No Do not know / cannot answer Photographs Yes Yes, to some extent No Do not know / cannot answer CD-rom, DVD Yes Yes, to some extent No Do not know / cannot answer

308 Seminars, conferences, workshops, other events Yes Yes, to some extent No Do not know / cannot answer Networks of experts Yes Yes, to some extent No Do not know / cannot answer Video/internet conferencing Yes Yes, to some extent No Do not know / cannot answer Round table discussions Yes Yes, to some extent No Do not know / cannot answer Newsletter Yes Yes, to some extent No Do not know / cannot answer TV Yes Yes, to some extent No Do not know / cannot answer Radio Yes Yes, to some extent No Do not know / cannot answer Newspaper Yes Yes, to some extent No Do not know / cannot answer Informal networking

309 Yes Yes, to some extent No Do not know / cannot answer Blogging, podcasting etc Yes Yes, to some extent No Do not know / cannot answer Internal training activities for members of organisation Yes Yes, to some extent No Do not know / cannot answer Other Yes Yes, to some extent No Do not know / cannot answer Total What are the target groups for the dissemination of your project(-s) results and experiences? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Your organisation (teachers, staff, learners etc.) Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Other target groups (if any): Education providers Associations Enterprises Public authorities Public service providers Counselling bodies Research centres Developers of informationcommunication technologies Non-governmental organisations Total

310 26. Did your dissemination and exploitation activities lead (or, are they likely to lead in the future) to the following? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Increased awareness of the quality, relevance and effectiveness of the results of your project by target groups of dissemination and exploitation activities Yes Yes, to some extent No Do not know / cannot answer Adoption and/or application of the results of your project by target groups for dissemination and exploitation activities within your organisation Yes Yes, to some extent No Do not know / cannot answer Adoption and/or application of the results of your project by target groups for dissemination and exploitation activities outside your organisation Yes Yes, to some extent No Do not know / cannot answer Transfer of the results of your project to appropriate decision-makers at regional (or local), national or European levels (not including reporting to national agencies) Yes Yes, to some extent No Do not know / cannot answer Adoption and mainstreaming of results into local, national or European policy on life-long learning Yes Yes, to some extent No Do not know / cannot answer Total Are you aware of good practice and project results of other organisations taking part in the LLP? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Yes, of many

311 Yes, of some No, not at all Do not know / can not answer Total Did you face (or, are you likely to face) the following obstacles affecting the dissemination, application and implemention of your project(-s) results? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Obstacles affecting dissemination of the outcomes of your project: Insufficient quality of the project results and other experiences Yes Yes, to some extent No Do not know / cannot answer Low concentration of dissemination activities Yes Yes, to some extent No Do not know / cannot answer It is difficult to reach appropriate target groups for dissemination Yes Yes, to some extent No Do not know / cannot answer Insufficient budget for using more or other dissemination instruments Yes Yes, to some extent No Do not know / cannot answer Not enough expertise for dissemination Yes Yes, to some extent No Do not know / cannot answer Insufficient staff expertise for virtual dissemination Yes Yes, to some extent No Do not know / cannot answer

312 Lack of motivation/interest/time in dissemination/exploitation within the organisation Yes Yes, to some extent No Do not know / cannot answer Other (please specify) Yes Yes, to some extent No Do not know / cannot answer Obstacles affecting the exploitation of the outcomes of your project (adoption or implementation of outcomes in policy or other projects): Resistance to change by certain target groups of exploitation Yes Yes, to some extent No Do not know / cannot answer Target groups of exploitation do not perceive disseminated practices as better than other practices disseminated at the moment Yes Yes, to some extent No Do not know / cannot answer Target groups of exploitation do not perceive disseminated practices as better than the current situation or products Yes Yes, to some extent No Do not know / cannot answer Disseminated good practices do not fit with the prevailing personal values of individuals Yes Yes, to some extent No Do not know / cannot answer Disseminated good practices do not fit with the prevailing legal regulations of the education system Yes Yes, to some extent No Do not know / cannot answer Lack of understanding of information-communication technologies (ICT) among the target groups* Yes Yes, to some extent No Do not know / cannot answer

313 Lack of ICT facilities in target organisations* Yes Yes, to some extent No Do not know / cannot answer Other (please specify) Yes Yes, to some extent No Do not know / cannot answer Total * - these answers were available only for beneficiaries of Transversal programme. 29. Are you aware that <sub-programme> programme is a part (sub-programme) of a larger EU Lifelong Learning Programme (LLP)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised Frequen Freque Frequen Freque Percen Freque Freque Percen cy ncy cy ncy t ncy ncy t Yes No Do not know / can not answer Total Are you aware of other sub-programmes of the EU Lifelong Learning Programme (LLP) (such as Comenius, Leonardo da Vinci, Erasmus, Grundtvig, Transversal, Jean Monnet)? Comenius Leonardo Grundtvig All Decntrl. Erasmus Decntrl. Decntrl. Decntrl. Centralised No, I am not aware of other sub-programmes Frequen cy Freque ncy Frequen cy Freque ncy Percen t Freque ncy Freque ncy Percen Yes, I am aware of the following sub-programmes: Comenius Leonardo da Vinci Erasmus Grundtvig Transversal Jean Monnet Do not know / cannot answer Total t 313

314 5.3. Survey of individual beneficiaries 1. You are (were during participation in the programme) a? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy A learner (pupil, student, trainee, adult learner) A teacher, trainer, researcher or other staff member An employee, employer or other staff member of enterprise , , , , , , , , , , ,8 18 1,1 77 2, , ,9 Other (please specify) 191 2,7 29 1,8 11,3 95 7,5 56 8,5 Total , , , , ,0 1b. Are you involved in the development of information-communication technologies or its content? (for teachers only) All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Yes , , , , ,7 No , , , , ,9 Do not know / cannot answer 168 2,4 86 5,2 40 1,2 17 1,3 25 3,8 Total , , , , ,0 2. Please assess the influence your participation in the <sub-programme> Programme had on the following: All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Your professional competences Strong influence , , , , ,6 Some influence , , , , ,6 No influence 548 7, , ,3 98 7,7 31 4,7 Do not know / cannot answer 270 3, , ,6 31 2,4 14 2,1 Communication in the mother tongue Strong influence , , , , ,9 314

315 Some influence , , , , ,3 No influence , , , , ,9 Do not know / cannot answer 261 3,8 71 4, ,5 48 3,8 23 3,5 Communication in foreign languages Strong influence , , , , ,1 Some influence , , , , ,9 No influence 368 5,3 82 5, ,6 79 6, ,8 Do not know / cannot answer 51,7 19 1,2 17,5 10,8 5,8 Digital competence (e.g. PC skills etc.) Strong influence , , , , ,9 Some influence , , , , ,9 No influence , , , , ,2 Do not know / cannot answer 175 2,5 43 2,6 71 2,1 38 3,0 23 3,5 Mathematical competence and basic competences in science and technology Strong influence 430 6,2 93 5, ,7 90 7,1 19 2,9 Some influence , , , , ,7 No influence , , , , ,8 Do not know / cannot answer 381 5, , ,8 70 5,5 46 7,0 Learning to learn Strong influence , , , , ,7 Some influence , , , , ,6 No influence , , , , ,2 Do not know / cannot answer 216 3,1 72 4,4 92 2,7 39 3,1 13 2,0 Social and civic competences Strong influence , , , , ,4 Some influence , , , , ,8 No influence 501 7, , , ,1 55 8,4 Do not know / cannot answer 116 1,7 38 2,3 50 1,5 18 1,4 10 1,5 Sense of initiative and entrepreneurship Strong influence , , , , ,2 Some influence , , , , ,1 No influence , , , , ,8 Do not know / cannot answer 195 2,8 64 3,9 83 2,5 32 2,5 16 2,4 Cultural awareness and expression Strong influence , , , , ,2 Some influence , , , , ,4 No influence 226 3,3 55 3,4 96 2,8 51 4,0 24 3,7 Do not know / cannot answer 65,9 22 1,3 25,7 11,9 7 1,1 315

316 Other (please specify) Strong influence 261 3,8 65 4, ,2 49 3,9 37 5,6 Some influence 70 1,0 17 1,0 28,8 18 1,4 7 1,1 No influence 111 1,6 33 2,0 50 1,5 18 1,4 10 1,5 Do not know / cannot answer 608 8, , , ,6 37 5,6 Total , , , , ,0 3. Which of the following language skills did your participation in the <sub-programme> programme help improve? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Did not help improving skills of foreign languages 275 4,0 70 4,3 78 2,3 63 5,0 64 9,7 Bulgarian 61,9 26 1,6 14,4 16 1,3 5,8 Croatian 19,3 3,2 14,4 1,1 1,2 Czech 161 2,3 34 2,1 83 2,5 36 2,8 8 1,2 Danish 73 1,1 14,9 49 1,4 5,4 5,8 Dutch 197 2,8 50 3,0 98 2,9 34 2,7 15 2,3 English , , , , ,4 Estonian 35,5 9,5 13,4 10,8 3,5 Finnish 176 2,5 36 2, ,3 22 1,7 7 1,1 French , , , , ,5 German , , , , ,5 Greek 127 1,8 41 2,5 57 1,7 14 1,1 15 2,3 Hungarian 118 1,7 35 2,1 47 1,4 22 1,7 14 2,1 Icelandic 31,4 9,5 16,5 4,3 2,3 Irish 30,4 5,3 16,5 7,6 2,3 Italian 593 8, , ,3 87 6,9 57 8,7 Latvian 37,5 15,9 12,4 8,6 2,3 Lithuanian 41,6 16 1,0 17,5 5,4 3,5 Maltese 17,2 2,1 4,1 7,6 4,6 Norwegian 67 1,0 16 1,0 45 1,3 6,5 0 0 Polish 293 4,2 75 4, ,0 61 4,8 23 3,5 Portuguese 223 3,2 36 2, ,6 22 1,7 8 1,2 Romanian 81 1,2 21 1,3 30,9 23 1,8 7 1,1 Slovak 88 1,3 22 1,3 28,8 18 1,4 7 1,1 Slovenian 56,8 22 1, ,1 6,

317 Spanish , , , , ,4 Swedish 214 3,1 56 3, ,7 22 1,7 12 1,8 Turkish 188 2,7 74 4,5 73 2,2 19 1,5 22 3,3 Other (please specify) 168 2,4 36 2, ,0 19 1,5 12 1,8 Total , , , , ,0 4. Please assess actual or possible career benefits (finding a job, promotion, financial or non-financial benefits) of your participation in the Programme. All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Strong career benefits , , , , ,4 Some career benefits , , , , ,1 No career benefits , , , , ,2 Do not know / cannot answer 640 9, , ,4 56 4,4 49 7,5 Total , , , , ,0 5. Would your project(-s)/activities have taken place and would you have participated in the project without funding from the <sub-programme>? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Most likely not , , , , ,5 Most likely yes, with the help of other national/international Most likely yes, with the help of other national/international Most likely yes, using own funds Most likely yes, using own funds, but with a lower intensity , , , ,7 48 7, ,6 72 4, ,2 54 4,3 21 3, ,1 74 4, ,1 82 6,5 33 5, ,5 71 4, ,3 70 5,5 29 4,4 Do not know / cannot answer 570 8, , , ,4 48 7,3 Total , , , , ,0 6. How would you assess your overall satisfaction with participation in the <sub-programme> programme? All Comenius Erasmus Leonardo Grundtvig 317

318 Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Very satisfied , , , , ,8 Rather satisfied , , , , ,6 Rather unsatisfied 134 1,9 30 1,8 80 2,4 19 1,5 5,8 Very unsatisfied 41,6 6,4 20,6 10,8 5,8 Do not know / cannot answer 42,6 15,9 17,5 5,4 5,8 Total , , , , ,0 7. To what extent were the activities of the <sub-programme> relevant to enhancing the following competences and changing the following attitudes: All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Creativity highly relevant , , , , ,6 rather relevant , , , , ,7 rather irrelevant , , , ,7 39 5,9 not relevant 423 6,1 49 3, ,4 96 7,6 29 4,4 Do not know / cannot answer 196 2,8 23 1, ,9 34 2,7 8 1,2 Competitiveness highly relevant , , , , ,9 rather relevant , , , , ,0 rather irrelevant , , , , ,3 not relevant , , , , ,2 Do not know / cannot answer 366 5,3 78 4, ,6 63 5,0 37 5,6 Employability highly relevant , , , , ,5 rather relevant , , , , ,7 rather irrelevant , , , , ,1 not relevant , , ,3 87 6, ,9 Do not know / cannot answer 640 9, , ,5 81 6,4 53 8,1 Entrepreneurial skills highly relevant , , , , ,5 rather relevant , , , , ,7 rather irrelevant , , , , ,5 not relevant , , , , ,3 318

319 Do not know / cannot answer , , ,6 87 6,9 55 8,4 Personal fulfilment highly relevant , , , , ,1 rather relevant , , , , ,3 rather irrelevant 286 4,1 92 5, ,5 53 4,2 22 3,3 not relevant 107 1,5 33 2,0 45 1,3 13 1,0 16 2,4 Do not know / cannot answer 92 1,3 35 2,1 38 1,1 15 1,2 4,6 Language learning highly relevant , , , , ,1 rather relevant , , , , ,5 rather irrelevant 307 4,4 74 4, ,2 82 6,5 43 6,5 not relevant 202 2,9 31 1,9 74 2,2 38 3,0 59 9,0 Do not know / cannot answer 71 1,0 12,7 34 1,0 13 1,0 12 1,8 Developing a sense of European citizenship highly relevant , , , , ,5 rather relevant , , , , ,8 rather irrelevant 541 7,8 76 4, , ,6 33 5,0 not relevant 248 3,6 34 2, ,6 46 3,6 13 2,0 Do not know / cannot answer 144 2,1 26 1,6 75 2,2 36 2,8 7 1,1 Total , , , , ,0 8. Did you actively promote (or are you likely to promote in the future) the experiences and skills you gained from participation in the <sub-programme> programme? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Yes , , , , ,7 Yes, to some extent , , , , ,1 No 214 3,1 35 2, ,9 40 3,2 6,9 Do not know / cannot answer 194 2,8 46 2, ,0 35 2,8 13 2,0 Total , , , , ,0 9. Have you heard from other people (colleagues, fellow students etc.) who have had good experiences with participation in the <sub-programme> Programme? All Comenius Erasmus Leonardo Grundtvig 319

320 Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Yes, of many , , , , ,6 Yes, of some , , , , ,7 No 332 4,8 84 5,1 50 1, , ,9 Do not know / cannot answer 100 1,4 27 1,6 25,7 31 2,4 17 2,6 Total , , , , ,0 10. One year before participation in the programme your level of awareness about the <sub-programme> programme was? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Sufficiently aware of the Programme Aware of the Programme to some extent Was not aware of the Programme , , , , , , , , , , ,0 84 5, , , ,3 Do not know / cannot answer 116 1,7 27 1,6 47 1,4 26 2,1 9 1,4 Total , , , , ,0 11. Are you aware that <sub-programme> programme is a part (sub-programme) of a larger EU Lifelong Learning Programme (LLP)? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Yes , , , , ,2 No , , , ,6 48 7,3 Do not know / can not answer 228 3,3 62 3, ,2 45 3,6 14 2,1 Total , , , , ,0 12. Are you aware of other sub-programmes of the EU Lifelong Learning Programme (LLP) (such as Comenius, Leonardo da Vinci, Erasmus, Grundtvig, Transversal, Jean Monnet)? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy 320

321 No, I am not aware of the other sub-programmes , , , ,3 61 9,3 Yes, I am aware of the following sub-programmes Comenius , , , ,5 Leonardo da Vinci , , , ,3 Erasmus , , , ,6 Grundtvig , , , ,9 Transversal 247 3,6 59 3,6 58 1,7 44 3, ,1 Jean Monnet 460 6,6 85 5, ,1 60 4, ,1 Do not know / cannot answer 95 1,4 27 1,6 43 1,3 14 1,1 11 1,7 13. What was your country of permanent residence (home country) during participation in the programme? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Austria 178 2,6 52 3,2 51 1,5 50 3,9 25 3,8 Belgium 140 2,0 25 1,5 62 1,8 35 2,8 18 2,7 Bulgaria 67 1,0 35 2,1 7,2 23 1,8 2,3 Cyprus 15,2 2,1 4,1 5,4 4,6 Czech republic 276 4,0 29 1, ,6 45 3,6 11 1,7 Denmark 75 1,1 30 1,8 37 1,1 6,5 2,3 Estonia 48,7 19 1,2 25,7 4,3 0 0 Finland 214 3,1 40 2, ,6 33 2,6 19 2,9 France , , , ,3 44 6,7 Germany , , , , ,5 Greece 32,5 5,3 16,5 5,4 6,9 Hungary 102 1,5 45 2,7 29,9 16 1,3 12 1,8 Iceland 15,2 2,1 8,2 5,4 0 0 Ireland 91 1,3 22 1,3 32,9 26 2,1 11 1,7 Italy , , , , ,0 Latvia 38,5 15,9 4,1 12,9 7 1,1 Liechtenstein 3, ,2 0 0 Lithuania 243 3,5 32 2,0 98 2,9 33 2, ,2 Luxembourg 3, ,0 1,1 1,2 Malta 27,4 9,5 7,2 4,3 7 1,1 the Netherlands 88 1,3 29 1,8 34 1,0 16 1,3 9 1,4 Norway 43,6 16 1,0 18,5 5,4 4,6 321

322 13. What was your country of permanent residence (home country) during participation in the programme? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Austria 178 2,6 52 3,2 51 1,5 50 3,9 25 3,8 Belgium 140 2,0 25 1,5 62 1,8 35 2,8 18 2,7 Poland 234 3,4 99 6,0 54 1,6 55 4,3 26 4,0 Portugal 139 2,0 49 3,0 63 1,9 6,5 21 3,2 Romania 169 2,4 51 3,1 53 1,6 52 4,1 13 2,0 Slovakia 86 1,2 23 1,4 37 1,1 20 1,6 6,9 Slovenia 166 2,4 10, ,1 11,9 7 1,1 Spain 587 8, , ,6 80 6,3 64 9,7 Sweden 142 2,0 30 1,8 74 2,2 23 1,8 15 2,3 Switzerland 5, ,1 1,1 0 0 Turkey 126 1,8 32 2,0 68 2,0 11,9 15 2,3 United Kingdom 485 7, , , ,9 56 8,5 Other (please specify) 31,4 4,2 16,5 10,8 1,2 Total , , , , ,0 14. What is your gender? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Female , , , , ,2 Male , , , , ,5 Total , , , , ,0 15. What is your age? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy up to 15 years 97 1,4 95 5,8 1,0 1, years , ,9 24, ,6 3, years ,2 78 4, , ,5 32 4, years , , , , ,7 65 and above 66,9 3,2 29,9 4,3 30 4,6 322

323 15. What is your age? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Total , , , , ,0 16. What is your educational background? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy Lower secondary education 248 3, ,5 21,6 59 4,7 12 1,8 Upper secondary education , , , ,7 31 4,7 Post-secondary non-tertiary education 522 7,5 90 5,5 76 2, ,7 43 6,5 Tertiary education, first stage , , , , ,4 Tertiary education, second stage , , , , ,3 Do not know / cannot answer 444 6, ,6 89 2, ,6 63 9,6 Total , , , , ,0 17. Do you belong to any of these groups? All Comenius Erasmus Leonardo Grundtvig Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy You are a person with special educational needs (e.g. disabled) You are considered to be socially and/or economically disadvantaged 104 1,5 31 1,9 35 1,0 24 1,9 14 2, ,7 34 2, ,3 47 3,7 28 4,3 None of these , , , , ,9 Do not know / cannot answer 266 3,8 60 3, ,0 57 4,5 14 2,1 18. If you were participating in mobility action, to which country (-ies) did the mobility take place during your participation in the <sub-programme> programme in the period ? All Comenius Erasmus Leonardo Grundtvig 323

324 Frequenc Frequen Frequenc Frequen Frequen y cy y cy cy I did not participate in a mobility action , , , , ,9 Austria 185 2,7 76 4,6 54 1,6 36 2,8 19 2,9 Belgium 638 9,2 49 3, , , ,5 Bulgaria 147 2,1 35 2,1 36 1,1 38 3,0 38 5,8 Cyprus 120 1,7 18 1,1 44 1,3 24 1,9 34 5,2 Czech Republic 183 2,6 72 4,4 77 2,3 23 1,8 11 1,7 Denmark 122 1,8 40 2,4 57 1,7 7,6 18 2,7 Estonia 152 2,2 26 1,6 78 2,3 27 2,1 21 3,2 Finland 236 3,4 67 4, ,1 14 1,1 16 2,4 France 298 4, , ,4 18 1,4 11 1,7 Germany 360 5, , ,6 52 4,1 28 4,3 Greece 290 4,2 59 3, ,2 68 5,4 55 8,4 Hungary 597 8,6 61 3, ,7 85 6, ,5 Iceland 161 2,3 19 1,2 75 2,2 18 1,4 49 7,5 Ireland 183 2,6 46 2,8 68 2,0 44 3,5 25 3,8 Italy 316 4, , ,4 19 1,5 9 1,4 Latvia 260 3,7 21 1, ,3 65 5,1 29 4,4 Liechtenstein 216 3,1 3,2 60 1,8 71 5, ,5 Lithuania 532 7,7 44 2, ,3 51 4,0 19 2,9 Luxembourg 80 1,2 5,3 69 2,0 5,4 1,2 Malta 191 2,7 21 1,3 67 2,0 65 5,1 38 5,8 the Netherlands 125 1,8 61 3,7 48 1,4 12,9 4,6 Norway 131 1,9 57 3,5 38 1,1 21 1,7 15 2,3 Poland 271 3, , ,0 24 1,9 14 2,1 Portugal 162 2,3 60 3,7 71 2,1 19 1,5 12 1,8 Romania 244 3,5 83 5,1 60 1,8 57 4,5 44 6,7 Slovakia 178 2,6 20 1, ,3 26 2,1 20 3,0 Slovenia 236 3,4 25 1, ,7 24 1,9 28 4,3 Spain 518 7, , ,6 12,9 18 2,7 Sweden 216 3,1 84 5,1 87 2,6 28 2,2 17 2,6 Switzerland 374 5,4 7, , , ,5 Turkey 258 3, ,0 47 1,4 25 2,0 39 5,9 United Kingdom 386 5, , ,0 17 1,3 2,3 Other (please specify) 180 2,6 24 1, ,5 14 1,1 22 3,3 324

325 ANNEX 6. THE LONGITUDINAL MAPPING OF THE LIFELONG LEARNING PROGRAMME 6.1. Comparison of the Previous Programmes and the Lifelong Learning Programme The LLP integrated four previous Community programmes: second phase of the Socrates ( ) programme, second phase of the Leonardo da Vinci ( ) programme, elearning ( ) multiannual programme and Community action programme to promote bodies active at European level and support specific activities in the field of education and training ( ). The latter programme was a predecessor of the current Jean Monnet sub-programme. At first glance it may seem that the integration of the previous programmes into the single LLP created a completely new programme in the field of lifelong learning. However, the longitudinal mapping of the LLP actions supported in the pre-2007 and post-2007 periods shows that majority of current actions have been already supported under the previous programmes. For instance, almost all activities financed under the current Transversal programme, which is often presented as a major innovation of the LLP, were supported (although to a lesser extent) under the previous Socrates, Leonardo or elearning programmes. Only the new action of multilateral networks aimed at encouraging language learning and linguistic diversity in Europe and financed under the KA2 of the Transversal programme was introduced as a totally new action. Also, a new group of various activities under the title of policy cooperation began to be supported under the KA1 of the Transversal programme aimed at promoting policy cooperation and innovation in lifelong learning, while all the other transversal actions, notwithstanding much more funding opportunities, remained nearly the same. However, it should be noted that the current Transversal programme allows supporting projects aimed at the beneficiaries from different sectors (e.g. higher education and vocational education and training) and ensures better visibility as well as more political attention for certain priorities such as multilingualism or dissemination and exploitation of results. Namely, the lack of dissemination and exploitation of results of predecessor activities were extensively criticised by the evaluation of the previous programmes. On the other hand, the launch of the LLP introduced several new actions, which were supplemented by new Programme lines in later years. Some of them clearly reflect the EU policy developments in the fields of education and training as well as EU-level priorities. First, more grants were offered to encourage individual mobility and more target groups were incorporated into the list of potential direct beneficiaries of the Programme. These new target groups include individual pupils under the new Comenius individual pupil mobility action (from 2010), staff from enterprises under the Erasmus staff mobility actions, higher education staff under the new Erasmus staff mobility for training action, individual adult learners under the new Grundtvig assistantships, workshops, senior volunteering projects actions and adult education staff under the new Grundtvig visits and exchanges for adult education staff action. Second, a clear emphasis on cooperation between higher education institutions and enterprises is seen from the introduction of a new corresponding Erasmus action or the incorporation of the previous Leonardo da Vinci student placement mobility action into the current Erasmus sub-programme, which contributed to a significant increase in the number of placements. Moreover, the new Erasmus staff mobility for training action allows staff from enterprises to spend a period of training or teaching in higher education institutions abroad. It should also be noted that besides encouraging cooperation between business and education providers, a new Comenius Regio partnership action, launched in 2009, show a clear priority to encourage the involvement of regional and local authorities into the cooperation activities in the school education sector. Third, the introduction of accompanying measures action in all the sectoral sub-programmes and KA2 of the Transversal programme could be seen as a way to ensure greater flexibility of the LLP, since it allows awarding grants to activities, which are not eligible to be supported under the sectoral programmes and the KA2 of the Transversal programme, but clearly contribute to the achievement of their objectives. Furthermore, the introduction of the new Erasmus Consortium Placement and Leonardo da Vinci Mobility Certificates reflect greater attention to the quality of mobility (besides its quantity) in correspondence to the European Quality Charter of Mobility. 325

326 Such new actions as Erasmus modernisation of higher education aimed at contributing to the modernisation of higher education or Leonardo da Vinci partnerships aimed at supporting small-scale cooperation between organisations working in the field of vocational education and training clearly reflect greater opportunities for the beneficiaries too. However, taking into consideration all other new activities introduced to the Programme, new actions constitute only a smaller part of the previously supported activities. Actions supported under the Jean Monnet programme, for example, remained totally unchanged. The longitudinal mapping of the LLP actions is provided in the table below. Table 6.1. The Longitudinal Mapping of the Lifelong Learning Programme Previous LLP Action LLP Action programme Category Comenius Socrates Preparatory visits Socrates Assistantships (assistants) Socrates Mobility Assistantships (host schools) Individual pupil mobility Socrates In-service training Socrates Multilateral school partnerships Socrates Partnerships Bilateral school partnerships Regio partnerships Socrates Mult. projects Multilateral projects Socrates Networks Multilateral networks Socrates Accompanying measures Accompanying elearning etwinning Erasmus Socrates Charter Erasmus University Charter Certificate Consortium Placement Certificate Preparatory visits Socrates Organisation of mobility Socrates Student mobility for studies Leonardo Student mobility for placements Socrates Mobility Staff mobility teaching Staff mobility training Socrates Intensive language courses Socrates Intensive programmes Socrates Curriculum development projects Modernisation of higher education Mult. projects Cooperation between higher education institution and enterprises elearning Virtual Campuses Academic networks Socrates Networks Structural networks Socrates Accompanying Accompanying measures Leonardo da Vinci Certificate Mobility Certificate Preparatory visits Leonardo Initial vocational training Mobility Leonardo People in the labour market Leonardo VET professionals Partnerships Partnerships Leonardo Transfer of innovation Mult. projects Leonardo Development of innovation Leonardo Networks Thematic networks Accompanying Accompanying measures Grundtvig Socrates Preparatory visits Visits/exchanges for adult education staff Assistantships Mobility Socrates In-service training Workshops Senior volunteering projects Socrates Partnerships Learning partnerships 326

327 Previous LLP Action LLP Action programme Category Socrates Mult. projects Multilateral projects Socrates Networks Multilateral networks Socrates Accompanying Accompanying measures Transversal Programme KA1: Policy Co-operation and Innovation in Lifelong Learning Socrates Mobility Study visits for EVT specialists Leonardo Socrates Observation Studies and comparative research Leonardo Mult. projects Networks Accompanying Policy cooperation KA2: Languages Socrates Leonardo Mult. projects Multilateral projects elearning Networks Multilateral networks Socrates Accompanying KA3: Development of ICT-based Content, Services, Pedagogies and Practice for Lifelong Learning Socrates Leonardo Mult. projects Multilateral projects elearning Socrates Networks Multilateral networks elearning Observation KA4: Dissemination and Exploitation of Results and Exchange of Good Practice Socrates Mult. projects Multilateral projects Socrates Observation Studies and reference material Leonardo Nat. projects Jean Monnet Pre-JM Mult. projects Multilateral research groups Pre-JM Chairs Pre-JM Chairs ad personam Pre-JM Centres of excellence Nat. projects Pre-JM Teaching modules Pre-JM Information and research activities Pre-JM Associations of professors/researchers Pre-JM Specified institutions Grants Pre-JM European Associations New actions Actions mentioned in the LLP Decision, but not supported Previous programmes: Socrates Second phase of the Community action programme in the field of education: Socrates ( ); Leonardo Second phase of the Community vocational training action programme: Leonardo da Vinci ( ); elearning Multiannual programme for the effective integration of information and communication technologies in education and training systems in Europe: elearning ( ); Pre-JM Community action programme to promote bodies active at European level and support specific activities in the field of education and training ( ). 327

328 ANNEX 7. ANALYSIS OF THE MONITORING DATA This annex looks at the monitoring data of the LLP. It provides an overview of the main indicators, concentrating on outputs and financial flows of the sectoral programmes (Comenius, Erasmus, Leonardo da Vinci and Grundtvig) and Transversal and Jean Monnet programmes. The Budget of the Lifelong Learning Programme The indicative financial envelope of the LLP was set at 6,97 billion. According to the available data, approximately 42% ( 2,95 billion) of this amount was allocated to the LLP sub-programmes during the analysed period. The annual breakdown of these allocations is provided in a table below (see Table 7.2). Our analysis suggests that Erasmus and Leonardo da Vinci programmes were the largest beneficiaries as they received the largest share of the allocated funds. In terms of funding distribution between centralised and decentralised LLP actions, about 9,7% of the total LLP budget was allocated to finance the implementation of centralised actions, meanwhile 85% of the remaining budget was earmarked for National Agencies implementing the decentralised actions (the remaining funds were used to organise specific calls for proposals, calls for tender and framework contracts, award the operating grants and finance other expenditures). Table LLP sub-programmes: budget increment during the period ( ) Comenius Erasmus Leonardo da Vinci Grundtvig Transveral Jean Monnet In total ,3% +37,1% +27,1% +11,5% +1,6% +0,7% 100% Analysis show that each year the amount of funding allocated to finance the implementation of the Programme was getting bigger. As a result, in 2009 the Programme budget was 15% larger than in 2007 (see Table 7.2). Most of this increment was due to larger allocations to the Erasmus programme. In 2009 the annual budget of this sectoral programme was 52 million larger than in 2007 (see Table 7.15). On the other hand, allocations to other sectoral programmes were getting bigger too. For example, financial allocations to Grundtvig in 2009 were 36% larger than in In fact, it was the most rapid growth between all four sectoral programmes, as the annual budgets of Comenius, Leonardo da Vinci and Erasmus were larger by 23%, 16% and 13% respectively. It should be emphasised that financial allocations to Transversal and Jean Monnet programmes largely remained at the same level. Table 7.2. Financial allocations for the LLP sub-programmes ( ) vs vs vs 2009 COMENIUS 16,1% 15,4% 17,2% Decentralised ,4% +15,2% +27,2% Centralised ,4% +61,9% +6,1% Subtotal ,4% +19,1% +23,1% ERASMUS 44,6% 46,1% 43,6% Decentralised ,9% 0% +11,9% Centralised ,0% +15,3% +32,6% Subtotal ,0% +0,7% +12,8% LEONARDO DA VINCI 25,9% 26,0% 26,0% Decentralised ,2% +6,7% +16,5% Centralised ,2% +6,3% +9,7% Subtotal ,9% +6,6% +16,1% GRUNDTVIG 4,9% 4,6% 5,8% Decentralised ,8% +60,1% +75,7% Centralised ,6% -2,0% -8,5% Subtotal ,0% +33,3% +36,0% TRANSVERSAL 5,6% 5,5% 5,0% 328

329 vs 2008 vs 2007 vs KA ,2% +9,8% +0,7% KA ,3% -31,0% +15,5% KA ,0% +3,9% -11,8% KA ,7% -57,5% +54,9% Subtotal ,9% -2,4% +4,3% JEAN MONNET 2,6% 2,2% 2,3% KA ,8% +11,3% +4,9% KA ,3% +2,3% +4,7% KA ,8% +1,8% 0% Subtotal ,4% +3,9% +4,3% OTHERS ,7% -41,8% -73,8% TOTAL operational expenditure ,4% +6,1% +15,0% Source: LLP Activity Report , Annex III Budget statistical data Note: All figures are ex-ante and rounded to 1 of their value. Comenius During the analysed period the Comenius share of annual LLP budget ranged from 15% to 17%, making it the third largest programme in terms of size of the budget. Most part of these funds was used to finance school partnerships and mobility grants to future teachers, teachers and other education staff (in-service training, assistantships). These trends might change to some extent in subsequent years as in 2009 and 2010 several new decentralised actions, namely Regio Partnerships and Individual Pupil Mobility, were introduced. It is planned that in initial stage the latter action should enable more than pupils to spend a period of studies in a partner school abroad. It should be noted that the monitoring data on individual mobility (pupils involved in joint activities supported by the LLP) is currently available only to a very limited extent. As a result, analysis of Comenius actions and their comparison is difficult. For example, although available data suggest that even relatively small increment of multilateral school partnerships budget can have a significant impact on numbers of pupils involved in activities of this action (comparison of the period figures), such findings are only indicative and should be confirmed by analysis of future monitoring data. The same applies to another finding that in order to increase the number of pupils involved in bilateral school partnerships a significant increase in funding is needed (no signs of increasing numbers of beneficiaries when funding remains at the same level). In the period Comenius supported almost assistantships for future teachers and more than in-service training courses for teachers and other educational staff. More than 73 million were allocated in order to attain such results and maintain the trend of increasing number of approved applications. In contrast, a sharp decline in the number of schools involved in multilateral school partnerships was evident in the same period. On the other hand, despite the scale of this change, it should be interpreted rather cautiously, because many applicants applied to prolong ongoing partnerships supported by the previous Socrates II programme in As a result, there is nothing odd in this sharp decline. In addition, schools are increasingly involved in bilateral partnerships. A notable amount of 57 million was dedicated to support the centralised Comenius actions in the period. More than 55% of these funds were used to support 117 multilateral projects. Meanwhile rest of the funds was allocated to finance etwinning activities, 13 multilateral networks and 12 other centralised projects. Although these actions involved around organisations, available funding supply did not match funding demand as only 1/3 of the applications submitted were successful in certain cases (i.e. Multilateral Projects and Multilateral Networks actions). 200 Table Comenius participants and financial allocations Action * Total Assistantships Applications received Statistics on Comenius centralised actions (Statistics Selected applications 2007, p. 8-9, Statistical report Comenius 2008, p. 16, Statistical report Comenius 2009, p. 17) 329

330 Action * Total (assistants) Applications approved Total grant amounts awarded Assistantships Applications received (host schools) Applications approved ,934 Applications received In-service training Applications approved Total grant amounts awarded Applications received n/a Applications approved n/a Multilateral school Total grant amounts awarded partnerships Number of schools involved Bilateral school partnerships Regio partnerships Multilateral projects Multilateral networks Accompanying measures Number of pupils n/a Applications received n/a Applications approved n/a Total grant amounts awarded Number of schools involved Number of pupils n/a Applications received Applications approved Total grant amounts awarded n/a n/a Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded e-twinning Total grant amounts awarded Source: European Commission and the Executive Agency. * Mobility data is based on the selected mobility numbers, i.e. data is not final. Erasmus Erasmus is the largest programme of the LLP in terms of budget size. On average, funds allocated to this programme amount to 45% (46% in 2008) of the annual budget of the Programme. Approximately 95% of these funds are earmarked to finance the decentralised mobility actions. Analysis of Erasmus mobility figures suggests that more than students had an opportunity to spend a period of studies abroad over past few years, including almost individual participants in Erasmus student mobility for studies (SMS) action and more than higher education students carrying out a placement abroad (SMP). The incorporation of previous Leonardo da Vinci II student placement mobility action into Erasmus programme substantially increased numbers of its beneficiaries. For example, the number of participants carrying out placements abroad, increased by 40% in 2007/2008, compared to the 2006/2007 academic year. Meanwhile the total number of students involved in both actions (SMS and SMP) increased by 5% in the same period. The most current data indicate that even more substantial, a 9% increment occurred next year (comparison of 2007/2008 and 2008/2009 academic year data). On the other hand, most of this progress is due to growing numbers of SMP beneficiaries. The LLP Decision set a quantified target of at least 3 million individual participants in student mobility actions under Erasmus and its predecessor programmes by The mark of two million students was reached in the academic year of 2008/2009. Although the number of students participating in Erasmus mobility actions has been steadily increasing since the launch of the programme in 1987 (with the exception of one academic year in 1996/1997), a shortage of approximately individual participants in order to reach the Erasmus target can be anticipated, unless the budget of Erasmus programme is constantly increased in order to sustain the growth of mobility figures (for more details see the simulation of three potential scenarios of further Erasmus programme implementation below, especially the first scenario which suggests that Erasmus quantified target can be achieved if SMS and SMP unit-costs remain at the 2009 level and the average growth of annual Erasmus budget is 7%). 330

331 The LLP introduced a new Erasmus staff mobility for training action, which offered a significant number of opportunities to train abroad in the academic year 2007/2008 and 2008/2009. Together with staff mobility for teaching, it allowed approximately exchanges, which is three and a half times higher compared to opportunities offered to Comenius staff in school education. Moreover, in the academic year of 2007/2008 and 2008/2009, more than individuals participated in Erasmus intensive language courses aimed at helping Erasmus students to prepare for studies or a placement abroad in less widely spoken languages. About students and teachers were also able to participate in Erasmus intensive programmes. Erasmus supported 185 projects under its centralised actions in the period, including 30 modernisation of higher education projects and 23 cooperation between higher education institutions and enterprises projects, which were supported under corresponding new actions. Overall, the number of projects supported under centralised Erasmus actions did not change a lot each year, except a sharp decline in selected projects under the virtual campuses action. Almost 56 million allocated to finance Erasmus programme projects in the period were primarily concentrated to finance academic and structural networks (around 18 million) and curriculum development projects (almost 14 million). 201 Table Erasmus participants and financial allocations Action 2007/ / /2010 Total Erasmus University Charter Number of institutions Number of grant agreements with Organisation of n/a higher education institutions mobility Total grant amounts awarded n/a Student mobility for Number of students n/a studies Total grant amounts awarded n/a Student mobility for Number of students n/a placements Total grant amounts awarded n/a Staff mobility for Number of participants n/a teaching Total grant amounts awarded n/a Staff mobility for Number of training periods n/a training Total grant amounts awarded n/a Intensive language Number of participants n/a courses Number of courses n/a 629 Intensive Number of participants n/a programmes Number of courses n/a Curriculum Applications received development Applications approved projects Total grant amounts awarded Modernisation of higher education Co-operation between higher education institution and enterprises Virtual campuses Academic and structural networks Accompanying measures Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Source: European Commission and the Executive Agency. Leonardo da Vinci 201 Statistics on Erasmus centralised actions (Statistics Selected applications 2007, p. 8-9, Statistical report Erasmus 2008, p. 31, Statistical report Erasmus 2009, p. 35) 331

332 In terms of budget size, Leonardo da Vinci is the second largest programme of the LLP, receiving around a quarter (26%) of the LLP budget each year. The largest part of these funds is usually spent on initial vocational training activities and transfer of innovation projects (see Table 7.20). Over the period Leonardo da Vinci offered an opportunity for approximately people to spend a period abroad, majority (62%) of beneficiaries being initial vocational training students. Due to the fact that a new Leonardo da Vinci Partnership action financed at the expense of previous mobility budget was launched in 2008, the mobility figures decreased from about participants in 2007 to only about in 2008, with the sharpest decline in target group of people in the labour market. On the other hand, in 2009 mobility figures have considerably improved (around participants). Therefore, it can be concluded that the decline was short-lived. Most of the aforementioned increase in mobility figures in 2009 should be attributed to the growing amounts of grants awarded to the initial vocational training action. The increase in funding consequently led to the growing numbers of participants. Meanwhile, the other two Leonardo da Vinci mobility actions every year faced funding cuts, but managed to produce a rather stable number of beneficiaries. Therefore, it seems that the budget decrease stimulated a better cost-effectiveness of these actions. The same conclusion could be drawn from available calculations on the expenditure per PLM and VETPRO participant ratio (see Table 7.19). Despite a notable increase in expenditure per participant in 2008 (probably due sudden cuts in funding and lack of time to adapt), the figures on expenditure ratio in 2009 were slightly below the 2007 level. Although the expenditure per IVT participant ratio did not follow this trend implicitly, the increase of expenditure ratio was minor only 55. Still, despite the increase in cost-effectiveness of Leonardo da Vinci mobility actions, the amount of funds allocated should have been considerably larger in order to reach the quantified target set by the LLP decision. With current funding only a little more than placements in enterprises were supported, which amounted to around 72% of the quantified target. Table Expenditure per participant in Leonardo da Vinci mobility actions ( ) Action Initial vocational training (IVT) People in the labour market (PLM) VET professionals (VETPRO) In the period 129 projects were supported under the centralised actions of the Leonardo da Vinci programme. A demand for the development of innovation projects has been steadily increasing together with the share of funds allocated for this largest centralised action. In contrast, the budget for thematic networks was decreased more than twice in 2009, with two times less thematic networks supported in 2009, compared with the previous year, despite a similar demand. Overall, the centralised projects involved over organisations in different Participating Countries. Still, the funds allocated to finance the centralised Leonardo da Vinci actions satisfied only a minor part of the demand. In case of development of innovation projects, only 19% of applications received during the whole period were successful. In case of thematic networks, also only 29% of the demand was satisfied. 202 Table Leonardo da Vinci participants and financial allocations Action * Total Applications approved Initial vocational training People in the labour market VET professionals Partnerships Number of participants Total grant amounts awarded Applications approved Number of participant Total grant amounts awarded Applications approved Number of participant Total grant amounts awarded Applications received Applications approved Number of training organisations involved 202 Statistics on Leonardo da Vinci centralised actions (Statistics Selected applications 2007, p. 8-9, Statistical report Leonardo da Vinci 2008, p. 16, Statistical report Leonardo da Vinci 2009, p. 17) 332

333 Action * Total Total grant amounts awarded Applications received Transfer of innovation projects Development of innovation projects Thematic networks Accompanying measures Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Source: European Commission and the Executive Agency. * Mobility data is based on the selected mobility numbers, i.e. data is not final. Grundtvig Grundtvig receives around 5% of annual LLP budget (5,8% in 2009) and is the smallest of all sectoral programmes in terms of budget size. In the period almost beneficiaries adult education staff were able to participate in Grundtvig mobility action for in-service training and carry out various activities in other European countries. Available figures indicate that the interest in this action is only increasing it is obvious from constantly growing numbers of received applications for staff mobility grants. The increment of allocated funds in 2008 has well reflected this trend and subsequently resulted in a significant increase of approved applications. Meanwhile in 2009, due to the introduction of two new staff mobility actions (Visits & Exchanges and Assistantships) there has been a further strong growth in numbers of submitted and approved applications. In addition, some changes in grant amounts awarded occurred. Like all other sectoral programmes, Grundtvig has a quantified target to achieve the programme should support the mobility of at least individuals involved in adult education per year by Figures provided in a table below suggest that up to 2008 Grundtvig was not on schedule for reaching its mobility targets. However, the introduction of four new mobility actions Visits and exchanges and Assistantships for adult education staff, Assistantships as well as Workshops and Senior volunteering projects for adult learners which were launched in 2009, has resulted in a substantial increase in numbers of individual mobility under Grundtvig programme, which now amounts to 88% of the target. Concerning the centralised actions, Grundtvig supported 202 multilateral project, 7 networks and 6 other projects, which altogether involved almost organisations from the Participating Countries.203 The most notable changes occurred within former action as the amount of funds allocated to finance multilateral projects and the number of approved applications was constantly decreasing during the analysed period. Table Grundtvig participants and financial allocations Action * Total Applications received Visits/exchanges for adult education staff Number of participants/ approved applications Total grant amounts awarded Applications received Assistantships Number of participants/ approved applications In-service training for adult education staff Workshops Total grant amounts awarded Applications received Applications approved Number of participants Total grant amounts awarded Applications received Applications approved Statistics on Grundtvig centralised actions (Statistics Selected applications 2007, p. 8,9, Statistical report Grundtvig 2008, p. 17, Statistical report Grundtvig 2009, p. 17). 333

334 Action * Total Number of participants Total grant amounts awarded Senior volunteering projects Learning partnerships Multilateral projects Multilateral networks Accompanying measures Applications received Applications approved Number of participants Total grant amounts awarded Applications received n/a Applications approved n/a Number of organisations involved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Source: European Commission and the Executive Agency. * Mobility data is based on the selected mobility numbers, i.e. data is not final. Transversal In terms of budget size, the Transversal programme is very similar to Grundtvig, as it receives approximately 5% of the annual LLP budget. Typically, the largest part of its funds (around 33% in the period) was allocated to KA2 action. As a result of the Transversal KA1 action, more than 710 study visits, which involved more than education and vocational training specialists and decision makers, were supported in the academic year 2007/2008 and 2008/2009. In addition to study visits, 19 grants for studies and comparative research were awarded to further policy cooperation and innovation in lifelong learning in the period. The KA1 of the Transversal programme also supported other policy cooperation activities, including the framework contract to provide consultancy services for the implementation of Work & Training 2010 work programme, grants for instruments and bodies aimed at supporting mobility and lifelong learning (Eurydice Network, Europass initiative, Euroguidance network, PLOTEUS portal, etc.), Presidency events, and specific calls for proposals. In terms of budget size, the largest key action of Transversal programme was KA2: Languages. This action also experienced the most significant budgetary increment of all Transversal actions. Although this increment was only temporary, the 2008 budget of KA2 became around 6% larger if compared to the previous year. The most part of additional funds was allocated to accompanying measures and multilateral networks. Overall, this key action supported 85 projects in the period, making it the largest single action devoted to languages. Meanwhile, the budget size of KA3 and KA4 actions decreased by around 16% and 33% respectively in the period. Available funds were used to support 56 KA3 projects and 36 KA4 projects. This certainly did not satisfy the overall demand for funding, as only a small part of applicants were awarded with grants despite their active participation (see the number of received applications). It was most evident in case of KA2, KA3 and KA4 multilateral projects. Table Transversal programme participants and financial allocations Action Total KA1: Policy Co-operation and Innovation in Lifelong Learning Study visits for VET specialists 2007/ / /2010 Number of participants 3687* 2616 n/a n/a Number of programmes 404* 306 n/a n/a Total grant amounts awarded * n/a n/a Applications received Studies and comparative Applications approved

335 Action Total KA1: Policy Co-operation and Innovation in Lifelong Learning research Total grant amounts awarded KA2: Languages Applications received Multilateral projects Multilateral networks Accompanying measures Multilateral projects Multilateral networks Multilateral projects Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded KA3: Development of ICT-based Content Applications received Applications approved Total grant amounts awarded Applications received Applications approved Total grant amounts awarded KA4: Dissemination and Exploitation of Results Applications received Applications approved Total grant amounts awarded Applications received 10 Applications approved 2 Total grant amounts awarded Source: European Commission and the Executive Agency. * Sum of study visits of the former Arion programme (Socrates II) and pilot phase of the merged Arion and Leonardo da Vinci study visits programmes in the 1 st semester of Total grant amounts awarded are only estimates. Studies and reference material Jean Monnet The Jean Monnet programme is the smallest programme of the LLP in terms of budget size as it receives less than 2,5% of the annual LLP budget. However, in 2009 the Jean Monnet programme involved 60 countries all around the world (number of countries in the Jean Monnet network). The KA1 of the Jean Monnet programme awarded grants to 118 selected chairs (including chairs ad personam), 34 centres of excellence and 124 teaching modules in the period. In addition, the number of applications for Jean Monnet Chairs, Teaching Modules, Information and Research Activities as well as most of other actions was constantly increasing. For example, a 70% increment was observed in 2009 in comparison to the previous year. Increasing numbers of submitted applications suggest that over time Jean Monnet programme brand is becoming better established and, therefore, the demand for additional funding is only increasing. The most part of Jean Monnet budget was allocated to the KA2 action, which supports operational and administrative costs of six specified institutions: the College of Europe (campus Bruges and campus Natolin), the European University Institute (EUI), the Academy of European Law (ERA), the European Institute of Public Administration (EIPA), the International Centre for European Training (CIFE), and the European agency for Development in Special Needs Education (AED). The EUI in Florence received the largest part of earmarked funds (more than 7 million per year), followed by the College of Europe receiving a yearly grant of more than 4,5 million. Funds of the KA3 action were used to support 56 European associations active in the field of education and training. Steady increment of grant amounts in this period corresponded with increasing demand for funding as the number of received applications also grew. Table Jean Monnet participants and financial allocations Action Total KA1: Jean Monnet Action Applications received Multilateral research groups Applications approved Total grant amounts awarded Chairs Applications received

336 Action Total Applications approved Total grant amounts awarded Applications received Chairs ad Applications approved personam Total grant amounts awarded Applications received Centres of Applications approved excellence Total grant amounts awarded Applications received Teaching modules Applications approved Total grant amounts awarded Applications received Information and Applications approved research activities Total grant amounts awarded Associations of Applications received professors and Applications approved researchers Total grant amounts awarded KA2: Operating Grants Supporting Specified Institutions Specified Applications received institutions Total grant amounts awarded KA3: Operating Grants to Support other European Institutions Applications received European Associations Applications approved Total grant amounts awarded Source: European Commission and the Executive Agency. Simulation of three potential scenarios of further Erasmus programme implementation In order to answer the operational question No. 5.1 of the evaluation, the analysis of three possible scenarios of further Erasmus programme implementation is presented below. All three scenarios seek to forecast to what extent the quantified target set by the LLP Decision shall be met by Two contradicting aims make it difficult to asses the Erasmus quantified target achievement. According to the available data, the annual budget of Erasmus programme shall remain at current level for the remaining duration of current Programme. Meanwhile, various calculations suggest that the current level of funding is not sufficient to achieve the quantified target unless the expenditure per Erasmus participant is decreased. On the other hand, there is a political pressure from the European Parliament to increase the amount of individual grant for Erasmus participants. As a result, at least three potential scenarios of further development are available. The first scenario is oriented towards the achievement of a quantified target. It is summarised in a table below. In order to meet the 2012 target of 3 million Erasmus students, certain conditions must be met. In particular, the SMS and SMP unit-costs should remain at the 2009 level and the number of SMP and SMS beneficiaries should increase by 6,66% and 3,4% each year respectively. These pre-conditions would result in Erasmus quantified target achievement with the average annual expenditure growth of 7%. Table Scenario I Meeting the quantified target without a unit-cost increase. Pre-conditions SMS* unit-cost ( ) SMP* unit-cost ( ) Participant number growth rate SMS participant number growth rate ,66% +3,4% Forecasting 2009/ / / /2013 In total Number of participants Number of SMS participants Number of SMP participants

337 SMS expenditure ( ) SMP expenditure ( ) Total expenditure ( ) Expenditure growth +7,0% +7,0% +7,0% +7,0% * SMS Student mobility for studies action, SMP Student mobility for placements action. The second scenario aims to achieve the quantified target in a combination with gradual increment of individual grants. In order to meet both of these aims, the following preconditions are necessary: each year the average number of SMS participants should increase by 3,4% and the average number of SMP participants should increase by 21,8%. These figures were obtained bearing in mind certain unit-cost changes (see the Preconditions part of the table). In fact, this scenario is very costly as on average the Erasmus budget should increase by 12% each year in order to meet such pre-conditions. In addition, this scenario is highly unlikely not only because the Erasmus budget size is not likely to increase in the subsequent years, but also because higher education institutions seek to increase the mobility figures rather than decrease the unit-costs. On the other hand, the unit-costs might start decreasing as a result of a current move towards the shorter study cycles of 3-year bachelors and 2-year masters instead of 4-5 year degrees. Table Scenario II Meeting the quantified target together with an increase in a unitcost Pre-conditions 2009/ / / /2013 SMS unit-cost ( ) SMP unit-cost ( ) SMS participant number growth rate +3,4% SMP participant number growth rate +21,8% Forecasting 2009/ / / /2013 In total Number of participants Number of SMS participants Number of SMP participants SMS expenditure ( ) SMP expenditure ( ) Total expenditure ( ) Expenditure growth +10,6% +12,4% +12,6% +12,4% The third scenario is based on the assumption that the total budget of each relevant action shall remain at the 2009 level for the remaining part of the current Programme duration. Moreover, it takes into a consideration the possibility that individual grants shall increase each year. In case of such scenario, a shortfall of approximately students should be expected by Table Scenario III Meeting the quantified target without additional funding and with an increase in a unit-cost Pre-conditions 2009/ / / /2013 SMS unit-cost ( ) SMP unit-cost ( ) Forecasting 2009/ / / /2013 In total Number of participants

338 Number of SMS participants Number of SMP participants SMS expenditure ( ) SMP expenditure ( ) Total expenditure ( ) Calculations of the average mobility unit-costs in sectoral programmes Outlined in tables below are the calculations on the average daily unit-cost of certain Erasmus, Leonardo da Vinvi and Grundtvig mobility actions. These calculations were used to answer the operational questions 16.1 and 16.2 of the evaluation. All of the analysed Erasmus mobility actions displayed the same trend their unit-costs were increasing in the period. Unit-costs of SMP action increased from 12,87 to 13,06, meanwhile, unit-costs of SMS action increased from 8,41 to 8,76. Daily costs of teachers and staff mobility were substantially higher than the mobility costs of learners, as they amounted to 111,23 in case of Staff mobility for teaching action and 115,54 in case of Staff mobility for training action. Student mobility for placements Student mobility for studies Staff mobility for teaching Staff mobility for training Table Average daily unit-cost of Erasmus mobility actions Number of participants Total grant amounts awarded Average duration Unit-cost per day 12,87 13,06 Number of participants Total grant amounts awarded Average duration Unit-cost per day 8,41 8,76 Number of participants Total grant amounts awarded Average duration 6 6 Unit-cost per day 107,93 111,23 Number of participants Total grant amounts awarded Average duration 6,5 6,5 Unit-cost per day 112,98 115,54 Learners Teachers and staff Quite different trends were evident in case of Leonardo da Vinci mobility actions. Although in the period the unit-costs of IVT, PLM and VETPRO actions were increasing, in 2009 they suddenly and quite significantly decreased. PLM and VETPRO actions even managed to increase the number of participants despite a decrease in awarded grant amounts. Therefore, it can be concluded, that the cost-effectiveness of these actions had increased. The same conclusion applies to the IVT action. Although the total grant amounts awarded for its participants increased in 2009, it has also resulted in a substantial increase in participant numbers. IVT PLM Table Average daily unit-cost of Leonardo da Vinci mobility actions Number of participants Total grant amounts awarded Average duration Unit-cost per day 45,78 52,64 47,51 Number of participants Total grant amounts Learners 338

339 VETPRO awarded Average duration Unit-cost per day 96,62 107,05 90,67 Number of participants Total grant amounts awarded Average duration Unit-cost per day 123,56 126,40 113,57 Teachers and staff In case of the Grundtvig programme, only one mobility action In-service training for adult education staff offers an opportunity to compare unit-cost changes over time. Calculations suggest that the unit-costs of this action were also rising every year. In comparison, the average daily unit-cost of each participant of this action has increased by 42,5 since In-service training for adult education staff Table Average daily unit-cost of Grundtvig mobility actions Number of participants Total grant amounts awarded Average duration Unit-cost per day 218,79 243,45 261,36 Teachers and staff 339

340 ANNEX 8. NOTE ON MONITORING THE LIFELONG LEARNING PROGRAMME The present note has been prepared in reaction to the requirement set in the Technical Specifications to propose a practical approach for both the continuation of the monitoring of the existing Programme and for the monitoring of the new generation of the education and training programme ; consideration should be given to the information needs of the Commission to support the execution of its main tasks. Introduction The purpose and the role of monitoring Monitoring can be defined as a "continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention (project, programme) with early indications of progress, or lack thereof, in the achievement of results". 204 Hence monitoring serves two equally important, closely interlinked purposes: Improving management (internal function; aims at making management of the programme more efficient); in this respect the key users of the monitoring information are managers at National Agencies, EACEA, and DG EAC; Strengthening accountability and transparency (external function; focused at providing empirical evidence on the outcomes of the EU co-financed activities and thus showing how EU intervention really creates value-added); in this respect the key users of the monitoring information are both inside the Commission (senior management at DG EAC; Secretariat-General; the Commissioners; etc.) and outside of it (representatives of the Participating Countries; Members of EP; etc.). The analysis underlying this note has revealed the mounting pressure to be in the position to provide adequate monitoring information to satisfy this particular need. As such the monitoring is closely interlinked with two other processes: Control; its key purpose is to ensure regularity and correctness of the intervention; the processes are especially closely interlinked in the so-called output-based control systems, where a project is paid for delivery of an agreed output, rather than reimbursed for the declared costs incurred during the project; in this respect control assists the monitoring function; Evaluation, which is judgement of interventions according to their results, impacts and needs they aim to satisfy; the key notion in this definition is that it is a process that culminates in a judgement (or assessment) of an intervention; the focus of evaluation is first and foremost on the needs, results and impacts of an intervention 205. There is a two-way relation: monitoring is expected to feed into evaluation some high quality monitoring information; on the other, evaluation results feed into the above mentioned external function of monitoring. For the purpose of this note we differentiate between these processes and focus mainly on monitoring. Levels of monitoring Additionally we should clearly differentiate between two key levels of monitoring since every of them presumes somewhat different needs for monitoring information: Project level monitoring; in the case of LLP this entails monitoring of the work of beneficiaries as implemented by the relevant management bodies: National Agencies for the decentralised actions; and EACEA for the centralised actions; the key focus here is usually on outputs, concrete deliverables to be produced by the beneficiaries in exchange for the granted support; Programme-level monitoring; DG EAC being in charge of the LLP should be in a position to monitor and report on the programme as the whole; the key focus here is on the core outputs, which could be 204 DG BUDG Publication on "Evaluating EU activities" July Ibid. 340

341 aggregated at the level of the Programme, but also results and impacts (i.e., the expected benefits which follow the production of the outputs and occur in the targeted groups but also beyond them). The key challenges in designing a monitoring system for a programme At least two general challenges are intrinsic to the design of any programme-level monitoring system: Balance between a general and specific approach to monitoring; there is a need to strike a balance between monitoring information which is specific to a particular area of intervention on the one hand (i.e., mobility vs. partnership vs. projects; primary/secondary education vs. VET vs. higher education vs. adult education; etc.), and the necessity of collating monitoring information which allows for aggregation and reporting at the level of programme as a whole on the other. In this respect, an important distinction can be made between monitoring at project level (where specific approach prevails) and programme-level monitoring. Meaningful assessment of what has actually been achieved through LLP implementation will only be possible through the analysis of information which can be aggregated and hence compared at the programme level. Importantly, such information will allow benchmarking and comparisons to be made between areas of intervention (including their supporting instruments) as well as across various types of outputs, or when relevant, stakeholders; Balance between factual (objective, schematic) information and knowledge-based (individual/expert) judgement; among the key functions of monitoring are the collection of data and its subsequent analysis/interpretation, so that raw data is turned into meaningful monitoring information which is useful for management (internal process improvement) and/or reporting (external accountability) purposes. To be useful such information has to be timely, reliable, relevant, and also neutral (i.e. schematic or objective ). To satisfy these criteria monitoring systems are commonly developed centred around the collection (and analysis/interpretation) of quantitative data on outputs, which can typically be directly linked back to the activities supported. In the case of LLP, examples of such data include the number and profile of participants, duration and direction of mobility, etc. However, there will also be a need to collect and interpret qualitative information in order to monitor the programme s implementation effectively. Monitoring of the Programme s results and impacts is impossible without qualitative analysis. While the primary purpose of monitoring is to obtain objective and neutral data (with evaluators then being asked to interpret monitoring data to determine what has been achieved, etc.), at the level of results and impacts, an element of judgement will be required in order to determine progress towards the Programme s aims (the feedback of mobility participants in the case of LdV is one of the few examples in the present LLP monitoring system). There is a common view that qualitative data is prone to be less reliable, harder to aggregate, compare longitudinally or use for benchmarking, its interpretation has inherent bias, etc. (i.e., such information is more subjective ). Yet, for example, structured feedback of participants, expert insights collected through focus groups and similar methods can be more useful for managers and decision-makers than mechanistic quantitative raw data. Therefore the assessment of success (e.g. extent to which LLP contributes to higher levels of entrepreneurship of its participants) will always contain an interpretative aspect. Subsequently, the importance of individual/expert judgement expert when monitoring and assessing actions implemented under the Programme shall not be underestimated. 1. Description of the current situation This chapter outlines the current situation concerning the availability of LLP monitoring data. It should be noted that monitoring data is not the same as monitoring information : raw/primary data coming from primary sources such as applications, progress/final reports, etc. needs to be analysed/interpreted to become monitoring information. Usually such analysis and interpretation is not performed as such but is rather encoded into monitoring indicators; e.g., availability of data on grant amount per project, origin (e.g., Member State) and type of beneficiary enables calculation of such indicators as total number of projects per Member State, average grant per Member State, etc. A good example of indicators which can be constructed from available monitoring data is the list of 10 indicators approved by the LLP Committee as a part of the document entitled Monitoring and evaluation of the Lifelong Learning programme 206 (see Table 8.1 below). 206 European Commission, DG Education and Culture, Monitoring and evaluation of the Lifelong Learning programme. Brussels (LLP/043/2010-Ann-1). 341

342 Table 8.1. Approved set of indicators for monitoring of LLP Number Indicator Statistics 1 Grants by consortia Number of CONSORTIA, grant per CONSORTIUM 2 Grants by partner Number of PARTNERS, grant per PARTNER 3 Grants per participant Number of PARTICIPANTS, grant per PARTICIPANT (Number of staff. Number of students / pupils / learners. Per country) 4 Consortium Composition and Size, by size, legal status Type of PARTNERS, HOSTS and HOMES (Public, Private, and commercial orientation etc.), per action. Per country. 5 Consortium Composition and Size, by type of Type of participating institutions (institution types, institution sector). Per country. 6 Number of CONSORTIA, number of PARTNERS per Consortium Composition and Size, Intensity of the consortia. Per COORDINATOR country. Per PARTNER cooperation and Country of the coordinator country. 7 From/To mobility Number of student / pupils / learners in-flows and outflows matrix. Per country. 8 Duration of the Mobility Number of participants in-flows and out-flows durations from days to months. Per country. 9 Educational Map of the Action Subjects, educational areas [ISCED], of the project. Per action. 10 Social Cohesion Topics addressed by the projects Social cohesion topics addressed by the projects (cultural diversity, xenophobia, special needs, equal treatment, sexual discrimination, racial discrimination, age discrimination). In the case of LLP, presently all monitoring data comes from applicants and project beneficiaries, the key sources being: - Grant applications (preliminary information, which is subject to changes during the course of implementation of the project); - Progress reports; - Final reports (by beneficiary and/or participant 207 ). It should be noted that there are two overlapping units of analysis for LLP monitoring data: - Data per project (most of the cases; also includes grants to individual mobility, as in these cases individual coincides with a project); - Data per participant (only these actions where participants mobility is organised by benefiting organisation, like Erasmus Mobility). Monitoring data available in Final reports presented by beneficiaries and participants is presented in table 8.2 below (it shall also be noted that availability of information, formulation of monitoring questions, etc. differs across the sub-programmes and actions). Table 8.2. Availability of monitoring data in project reports and participant reports Data is necessary for Possibility to aggregate data Number Data monitoring 10 agreed indicators Information on beneficiaries and partners (type of 1 organization, commercial orientation, scope, legal status, size (staff, students/ pupils/ learners), sector, contact data, etc.) Harmonisation of data across the subprogrammes and (or) actions Financial data (EU funding per project/ partner, etc.) Number of participants (no. of students/ pupils/ learners) and type of participants (educational background, study circle, age range, occupations, employment status, gender, special needs, etc.) From/to mobility (student/ pupil/ learners in-flows and out-flows in numbers and mobility duration) 207 Note: only in the case of LdV, participant s final report also includes the feedback from a participant. 342

343 Number Data Data is necessary for monitoring 10 agreed indicators Possibility to aggregate data Harmonisation of data across the subprogrammes and (or) actions 5 Educational/training fields Horizontal issues (special needs, gender equal treatment, xenophobia, racial discrimination, cultural diversity, etc.) addressed by the project Relevance to the operational objectives of the programme 8 Working languages Key competences addressed by the project - + +/- 10 Description of outputs and deliverables - - +/- 11 Description of impacts, impacts on organisation - - +/- 12 Level of achievement of results/ products/ outcomes Level of achievement of aims/objectives Dissemination of project results (types) - + +/- 15 Dissemination of project results (description) - - +/- 16 Information on participants (primary source: final participant reports): satisfaction levels (on different subjects), impact on key qualifications, perceived impact on employment status and occupational improvements, etc /- In the table, Possibility to aggregate data refers to availability of data across LLP sub-programmes and actions (i.e., whether such or similar data being collected and whether it is possible to quantify and aggregate data across different projects/actions). Of course, to ensure correct aggregation data shall be harmonised/ comparable, e.g. it should be ensured that data definition is the same across the actions. Information presented in reports by beneficiaries (and participants) can be and is used for project level monitoring (by National Agencies for decentralised actions and Executive Agency for centralised actions). Meanwhile, monitoring at programme level requires possibilities for aggregation of data. In turn this means that data in project and participant reports must be available and harmonised (comparable) across different subprogrammes and actions. Also, to minimise administrative workload preferably there should be IT tool to assist data collection, aggregation and reporting. Availability and harmonisation of data across different subprogrammes and action types and availability of suitable IT tools to collect and process available data provides for possibilities to seamlessly aggregate and analyse various monitoring information. The availability of monitoring data and supporting IT tools is fragmented across the programme. E.g. Rap4Leo mobility tool provides possibility to generate monitoring data for Leonardo da Vinci mobility actions both at project beneficiaries and individual applicant s level, which is not available under other sub-programmes. Moreover, the situation has changed over the years ( ). In the case of decentralised actions, recent introduction of LLPLink and its e-forms (first, Application eforms, and more recently Reporting eforms) will in principle change scope of available monitoring data generally all monitoring data presented in reports of project beneficiaries (subject to proper adjustments and harmonisation of data fields) can be processed and used for programme monitoring purposes (for those actions where Reporting eforms will be developed). There are the following Application eforms: - Comenius Assistantships (Assistants and Hosts Schools); - In-service Training (Comenius and Grundtvig); - Erasmus Mobility; - Leonardo da Vinci Mobility (Initial Vocational Training, People in the Labour Market, VET Professionals); - Leonardo da Vinci Transfer of Innovation; - Partnerships (Comenius Multilateral and Bilateral, Grundtvig and Leonardo); - Preparatory Visits (Comenius, Erasmus, Leonardo da Vinci, Grundtvig). 343

344 They cover the absolute majority of projects funded under decentralised actions. For example, the total number of valid 2010 applications is Total number of applications submitted for the actions served by these 7 eforms is ; this represents 95% of the total number of applications for decentralized actions (without study visits). The total number of eforms actually submitted is This represents 80% of the total number of applications for decentralized actions (without study visits) and 84% of the actions served by eforms. The following Reporting eforms are already being used: - Comenius Assistantships (Assistants and Hosts Schools); - In-service Training (Comenius and Grundtvig); - Partnerships (Comenius Multilateral and Bilateral, Grundtvig and Leonardo). The ones currently under development and still to be introduced include: - Leonardo da Vinci Mobility (Initial Vocational Training, People in the Labour Market, VET Proffessionals); - Leonardo da Vinci Transfer of Innovation. Finally, LLPLink features a system of aggregated reporting templates to facilitate access, query and analysis of monitoring data contained in the system. Analysis of monitoring data which is being collected in project reports and participant reports (see Table 8.2 above) shows that data necessary for calculation of 10 agreed monitoring indicators in principle is available, data is harmonised across the sub-programmes and actions of LLP and it is possible to aggregate this data. However, as it takes times to populate LLPLink with relevant data, the monitoring data from project reports will be available for aggregation only from ; during its aggregation process, it will be vital to test the accuracy of the data and to make relevant improvements (adjustments of reporting eforms, etc.) if needed to ensure consistency and reliability of data on achievement of LLP indicators. Further the analysis shows that certain data from monitoring reports which in principle is available and harmonised across the sub-programmes and actions of LLP is not being used for calculation of LLP indicators, therefore new indicators could be introduced (e.g. information on working languages of the projects, new indicators could be related to increase in linguistic diversity, etc.). Planned Mobility tool will further expand the potential of monitoring data collection and aggregation. It will contain data per participant for the actions where participants mobility is organised by benefiting organisation: Erasmus Mobility and Leonardo da Vinci Mobility. Potentially available and processable monitoring data will include data on impact of executed mobility on participants key qualifications, perceived impact on employment status and occupational improvements, etc. (currently this kind of data is presented in participants reports). In the case of centralised actions, Saykiss plays an important role in assisting the data collection and analysis. The system features Application eforms for a number of actions; yet there are no Reporting eforms at all. There is a theoretical possibility for DG EAC staff to access, query and analyse data contained in Saykiss database directly, but that requires knowledge of Business Objects on behalf of a user (a specialised query, reporting and multi dimensional data analysis tool). Due to this EACEA primarily provides already aggregated data: regularly (according to the agreed templates), but also ad-hoc (based on specific needs raised by DG EAC). 2. The need for and use of monitoring information (internal/management needs and external/accountability needs) The need for and use of monitoring information for internal/management needs As mentioned before, available monitoring data is primarily used by National Agencies and EACEA for management of projects under decentralised and centralised actions. The analysis revealed that at the level of DG EAC the following monitoring information is most frequently needed for Programme management purposes: Budget distribution/ amounts awarded (by country, sub-programme, action type, etc.; Application (total no. and selected ones); Grant agreements concluded (no. and amounts); Mobility from/to (no. and duration); 208 As on June This figure includes all actions with the exception of Study Visits, also those actions for which no eform has been developed. Application rounds for preparatory visits and in-service training are still open for

345 Payments. As indicated above main monitoring information regularly used for internal DG EAC management needs is therefore related to financial and operational management data. The analysis indicates that this type of data is already largely available. Moreover, access to it for DG EAC will improve pursuant to introduction of LLPLink and its eforms. It was also noted, that for internal management purposes but especially for external accountability needs the available monitoring information (related to operational issues and outputs) is further complemented with findings stemming from evaluations, which are carried out with respect to different LLP implementation aspects (impact of the LLP or the LLP sub-programmes on certain policy area, etc.) and provide information on results and impacts of the programme. The need for and use of monitoring information for external/ accountability needs As it is shown above the overall system of LLP monitoring even at the programme level is geared at monitoring inputs (e.g., budgets) and outputs (e.g., mobility numbers). This trend was established still by the LLP decision itself, which contains four quantified targets, related to expected outputs of four sectoral programmes of the LLP: Comenius: involvement of at least 3 million pupils in joint educational activities over the period of the programme; Erasmus: contribution to the achievement by 2012 of at least 3 million individual participants in student mobility under the Erasmus programme and its predecessor programmes; Leonardo da Vinci: increase placements in enterprises to at least per year by the end of the programme; Grundtvig: support the mobility of at least individuals involved in adult education per year by The system of the LLP monitoring was further developed by a special working group on impact monitoring, which provided its proposals for the LLP monitoring framework and indicators. Based on these proposals, the system of LLP monitoring indicators was approved by the LLP Committee in June 2010 (see Table 8.1 in the chapter above). The set of indicators and established LLP monitoring framework will allow generation of monitoring information that is necessary to report on the quantified targets defined in the LLP Decision. However, all approved indicators are limited to measuring outputs, there are no result and/or impact level indicators among them. 3. Areas for possible improvement and strengthening of the baseline situation The analysis revealed that the present monitoring system is largely sufficient for generating monitoring information: Concerning the four quantified targets set in the LLP Decision; Required by majority of DG EAC staff for daily programme management needs (some needs which are specific to certain sectoral programmes or key activities can be further analysed). At the same time the present monitoring system is not geared towards monitoring results and impacts. This is quite in contrast to ever more popular tendency to emphasise results-based management approach to managing public intervention programmes. Given the magnitude of the challenges to be tackled by the LLP, its relatively limited budget and wide scope of the programme, successful implementation of such a programme (measured in terms of effectiveness and impact) depends on a shared understanding of its role and objectives and a clearer focus on activities where the EU has the strongest added value. To respond to such management challenge, the public authorities tend to emphasise results-based management approach, the strategy which focuses on the achievement of results. In particular, it emphasises the importance of defining expected results (i.e., how do we define success for the Programme?) with the involvement of key stakeholders, assessing the risks that may impede expected results, monitoring the programme designed to achieve these results through the use of appropriate monitoring measures, reporting on performance in achieving results, and acting on performance information. This represents a fundamental reorientation away 345

346 from previous management approaches that were dominated by an emphasis on inputs and activities, the assumption being that results would follow if the inputs and activities were appropriately robust. Regular monitoring of programme results is an integral part of this approach and a key component of the informed management and the identification of opportunities for improved public sector performance. It also serves a function of accountability. Such an approach focuses on the outcomes (i.e., results and impact) of public expenditure so as to understand the benefits of implementing various actions and making these clear to European citizens. A schematic illustration of the main steps involved in results-based management is provided in Figure 8.1. Figure 8.1. Main steps of results-based management PLANNING IMPLEMENTATION MONITORING REPORTING Learning and adjusting Source: Treasury Board of Canada Secretariat, Results Based Management see < This is not to say that similar processes are not taking place in DG EAC when managing the LLP. Still these processes are rather fragmented (e.g., each sectoral programme tends to understand the success of the programme in some specific way) and do not contribute to the overall integrated approach to the programme and its management. Therefore a possible area of improvement lays in expanding the present monitoring system to incorporate more systematic monitoring of results and impact. To achieve this one would have to: Develop and agree upon the logic of intervention; it should be noted that the abovementioned multiplicity of objectives, which frequently is the outcome of political compromise (i.e., the oval in Figure 8.2 below), does not necessarily preclude from establishing a more working version of logic model (i.e., the rectangle below), which serves as a key tool to explain what the programme is about and what it aims at (i.e., how do we define success for some of the key parts of the programme?), and Then serves as a reference point to further developing monitoring system. 346

347 Figure 8.2 Objectives set in the Decision Impacts (what are longer term benefits occurring to systems which are aimed at? How do we define success?) Results (what are immediate benefits occurring to target groups: individuals and organisations? How do we define success?) Inputs (budget) Implementation Outputs (Types: what is delivered by the programme? How do we define whether outputs are of quality?) 4. An outline for a monitoring system of the current programme and for the successor programme As indicated in the previous chapters, LLP monitoring framework is oriented at monitoring outputs yet the impact monitoring framework is very limited. Defined LLP monitoring indicators are largely sufficient for regular management of the Programme at output level and for generation of monitoring information that is necessary to report on the quantified targets defined in the LLP decision, yet they are not suitable for results and impact monitoring. The key component of any monitoring system is the logic of intervention, which reveals at least schematic causal relations between the funded actions and intended effects. Such logic can take a number of forms, such as: Logic model (also known as objectives tree ), Hierarchy of limited number of objectives (containing quantified targets, where feasible), or similar, and it serves a key point of reference for both designing the monitoring system during programming phase and later for its assessment and constant improvement. In the case of LLP, the large number of objectives (4 general objectives of the LLP, which are supplemented with 13 specific objectives and 33 operational objectives of LLP sub-programmes), which partially overlap (please see relevance evaluation in the Interim evaluation report), makes it difficult to establish clear LLP impact monitoring framework. As it was described above, monitoring information presented by project beneficiaries and participant in their reports (please see Table 8.2 in the chapter above) already contains monitoring information, which could be used to assess the impact off LLP, therefore development of logic model (please see third chapter of the note) could be combined with development of result and impact indicators (the list below was generated by the evaluator and is presented for illustrative purposes only): Higher levels of lifelong learning: - (increase in) share of LLP-participants in CVT in relation to all CVTS participants; - Share of LLP-participants in relation to all participants of a certain age group (e.g , , 65+); - LLP-motivation index: number/share of LLP-participants with repeated mobility activities (during ET, job, excl. private/vacation purposes); - LLP-creativity index: number/share of patents by LLP-participants. 347

348 Fostering interchange and mobility (between education and training systems in the EU): - Share of outgoing and incoming LLP-students/LLP-participants to all mobile students/persons for ET. - Share of new LLP-participants who were motivated by former LLP-participants (could also be broken down by subprogramme). - Number/share of participants with repeated mobility activities (during ET, job). Contribution to the development of quality LLL and promotion of high performance, innovation and a European dimension in systems and practices in the field: - Number/share of innovative practices taken over by other institutions/countries after participation in LLPactivities. Realisation of European area for LLL: - (Increase in the) share of outgoing/incoming LLP-participants in relation to all outgoing/incoming people (for ET purposes only). Gender equality: - LLP gender equality in Math, Science and technology (in relation to general gender distribution in MST); - LLP gender equality in Arts (in relation to general gender distribution in MST). Personal fulfilment: - Job satisfaction of LLP participants in relation to job satisfaction of non-participants. - Increase in job satisfaction of LLP participants in relation to job satisfaction prior to participation. Employability: - Unemployment rate of LLP-participants in relation to overall unemployment rate; - Increase in the number of employed people, reduction in the share of unemployed people, transition rates into the labour market (within a certain period) after training; - Growth of an entrepreneurial spirit; increase in number of new or newly firms by former LLP-participants, growth rates (in turn over) of companies founded by former LLP-participants within the last two/five years. Language learning: contribution of LLP to language learning in a MS - Linguistic diversity: increase in the (average) number of languages spoken by individuals; average number of languages spoken by LLP participants (in relation to non-participants). - Number of languages learnt after participation in the LLP. As it was mention in the first chapter of the note, the main data sources of monitoring information are reports presented by the project beneficiaries, however monitoring at programme level requires possibilities and mechanisms for collection (preferably automated collection) and aggregation of data. LLPLink reporting eforms in principle change the scope of (potentially) available monitoring data, however practical use of LLPLink data will remain subject to further adjustment and harmonisation of data fields across different actions and subprogrammes. The planned Mobility tool will further expand the potential of monitoring data collection and aggregation, potentially available and processable monitoring data will include data on impact of executed mobility on participants key qualifications, perceived impact on employment status and occupational improvements, etc. Analysis of monitoring data which is being collected in project reports and participant reports (see Table 8.2 in the first chapter of the note) shows that data necessary for calculation of 10 monitoring indicators in principle is available, data is harmonised across the sub-programmes and actions of LLP and it is possible to aggregate this data. As a rule, such data will become available from onwards. Also, the analysis revealed that certain data from monitoring reports in principle is available and harmonised across the sub-programmes and actions of LLP, yet it is still not being used for calculation of LLP indicators, therefore new indicators could be introduced if required (e.g. information on working languages of the projects, new indicators could be related to increase in linguistic diversity, etc.). As for LLP successor programme, the logic model of programme intervention (i.e., the model which would clearly set out what benefits are expected to accrue to target groups and beyond as a result of implementing the projects and delivering the outputs; as such it would describe in a more structured way what is perceived as 348

349 a success of the programme from the manager s point of view) and system of indicators shall be developed and agreed at programme design stage. Reporting forms and IT support tools shall be then designed with a view to collect from projects beneficiaries and process the information necessary for programme impact monitoring needs. Development of logic model of Programme intervention and results and impact monitoring framework at programme design stage would allow for more evidence-based policy and for deployment of IT support tools and automated collection of monitoring data. 349

350 ANNEX 9. LIST OF REFERENCES AND SOURCES European Commission Commission of the European Communities, An updated strategic framework for European cooperation in education and training, COM(2008) 865 final. Commission of the European Communities, European benchmarks in education and training: follow-up to the Lisbon European Council. Available online at: Commission of the European Communities, Final report on the implementation and impact of the second phase ( ) of the Community action programmes in the field of education (Socrates) and vocational training (Leonardo da Vinci) and the multiannual programme ( ) for the effective integration of information and communication technologies (ICT) in education and training systems in Europe (elearning), Brussels, , COM(2009) 159 final. Commission of the European Communities, Improving competences for the 21st Century: An Agenda for European Cooperation on Schools, Brussels, , COM(2008) 425 final. Commission of the European Communities, Multilingualism: an asset for Europe and a shared commitment, COM(2008) 566 final. Commission of the European Communities, Progress towards the Lisbon objectives in education and training, COM(2009) 640. Commission of the European Communities, Progress towards the Lisbon objectives in education and training, COM(2009) 640 final. Commission of the European Communities, Promoting the learning mobility of young people, Green Paper, COM(2009) 329 final. Commission staff working paper, A Memorandum on Lifelong Learning, SEC(2000) 1832, Brussels, Commission staff working paper, Proposal for a Decision of the European Parliament and of the Council for an integrated action programme in the field of lifelong learning. Extended impact assessment integrating ex-ante evaluation requirements, SEC(2004) 971. Communication from the Commission to the Council and to the European Parliament Efficiency and equity in European education and training systems, COM(2006) 481 final. Communication from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions, Making a European Area of Lifelong Learning a Reality (COM(2001)678 final). Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions on 16 December New Skills for New Jobs: Anticipating and matching labour market and skills needs. COM(2008) 868 final. Dissemination and exploitation of the results, retrieved 19 August 2010 from ttp://ec.europa.eu/education/lifelong-learning-programme/doc98_en.htm. European Commission, Final report on the implementation and impact of the second phase ( ) of the Community action programme in the field of education (Socrates) and vocational training (Leonardo da Vinci) and the multiannual programme ( ) for the effective integration of information and communication technologies (ICT) in education and training systems in Europe (elearning), COM(2009) 159 final. European Commission, Comenius School Education - Creativity And Innovation: European Success Stories (2009). European Commission, Lifelong Learning Programme: statistical Overview of the Implementation of the Decentralised Actions in the Erasmus Programme in 2007/