Annex 1 Stichting Woord en Daad T.a.v. Wouter Rijneveld Postbus 560 4200 AN Gorichem WOLF; Milieuadvies & Conflictbemiddeling Cornelis Trooststraat 62 hs 1072 JH Amsterdam KvK: 343315859 Tel. 0643087687 wolf@wolfgangrichert.nl Proposal for the evaluation of the Northern component of Woord en Daad s Policy Influencing programme 2008-2012 5 March 2013 Introduction In its Terms of Reference Woord en Daad provides concise information about the programme for Policy Influencing with both a Southern and a Northern component. Woord en Daad is looking for independent consultants to evaluate both components separately. This is a proposal for the evaluation of the Northern component. For Policy Influencing in the North Woord en Daad wants an evaluation of the activities from their start in 2008 up to the end of 2012. The Policy Influencing in the North programme consists of at least 16 trajectories. The seven most relevant trajectories were selected to be the sample for this evaluation: 1. Lobby for Technical and Vocational Education and Training (TVET) on national and EUlevel. 2. Lobby for small and medium enterprise (SME) development/value chain development. 1
3. Lobby for education. 4. Lobby for the Colombia Platform. 5. Lobby on the Coordination Europe-Haiti. 6. Unforeseen lobby activities. 7. Lobby with Partos. Purpose of the evaluation The main reason to carry out this evaluation is to get insight in the choices that Woord en Daad made and the different approaches that are used. The outcomes of the evaluation will be used by Woord en Daad for the future policy and further choices on how to support partners and how to realize PI objectives. Primary intended user(s) and use(s): 1 This proposal for an evaluation process is designed around the needs of the primary intended users. In general, the users are those particular individuals or groups who intend to use the evaluation process or findings to inform their decisions or actions. This is not the same as the intended audience, which has a more passive relationship with the findings of an evaluation. The primary intended users of this evaluation are the key advocacy staff and the management from Woord en Daad. The organization intends to use the evaluation findings and learn lessons for future policy decisions. Secondary users are Woord en Daad s Southern partners. They are indirectly involved also in the lobby in the North and Woord en Daad assumes that the evaluation findings will be of interest for them as well. Therefore, attention will be given to the connection between the Northern and the Southern programme. The users intentions will best be served if they are, to some extent, involved in the evaluation process itself. This starts with discussing some of the proposed choices regarding the evaluation methodology as these are not rocket science but choices. Next, the users will be involved in the explicit reconstruction of their theory of change and the description of the achieved outcomes, preferably as team, which maximizes the learning. Finally, a joint discussion of the evaluation findings will be facilitated. This involvement typically results in increased use of the evaluation. 1 Inspired by Ricardo Wilson-Grau who based his thoughts on Michael Quinn Patton s Utilization-Focused Evaluation, 4 th Edition, 2008. 2
Evaluation questions In its Terms of Reference Woord en Daad distinguishes between descriptive and evaluative questions to be answered. The descriptive questions will be answered by Woord en Daad staff themselves leading to extended reports of seven trajectories and shorter reports/documentation of nine trajectories. Descriptive questions - Why did WD initiate a Northern component of the Policy Influencing programme? - Which methods/strategies have been used and why/how? And which resources were used? - What cooperation took place with other organizations/institutions? And what were the reasons to start/continue this cooperation? - A short case description for each trajectory: o Why were certain trajectories successful? o What influenced the length of trajectories? o What were the key events of the trajectories? o What are the results? Evaluative questions Relevance 1. Can the choices of Woord en Daad be related to the political context? Were certain opportunities to connect to the political context left unexploited? 2. Do the activities offer added value over what is offered by other lobby actors? 3. What choices have led to the focus within and between the lobby trajectories and why were certain trajectories started? Is there sufficient balance in the width and depth of the trajectories? 4. Was there a clear connection between the PI trajectories in the North and the identified needs in the South? Effectiveness 5. How effective have the trajectories/programmes been? What are the explanatory factors for the effectiveness (or lack of effectiveness) of trajectories/the programme? Methodology for this evaluation The proposed methods of investigation and analysis ensure that the evaluation s findings are credible to primary intended users. The methodology is feasible. The evaluation procedures are practical and cost/effective with the minimum disruption of activities to achieve the maximum required information. The methodology will also be ethical, taking 3
into account the welfare of those involved in the evaluation, as well as those affected by its results. 2 Development of a specific policy regarding confidentiality is not foreseen. The three key methodological tools for this evaluation will be a reconstruction of the Theory of Change, outcome harvesting and substantiation with external informants through in-depth interviews and a survey. They will be used for each of the 7 selected trajectories and will be implemented parallel. I assume that it will be possible and logical to make combinations. For example, some theories of change might have overlap with each other, some internal staff will be knowledgeable about more than one trajectory and so will some external informants. In all those cases, smart combinations will be sought. Step 1: Reconstruction of Theory of Change In the world of politics, development and social change, the use of theories of change has gained currency, particularly on the part of donors, as they become aware that the environment in which they work is complex. This is the case of policy influencing by Woord en Daad. They face so many variables and factors interacting that they have little certainty about whether what they do will lead to the results they seek; i.e., the results of their activities are unpredictable. In addition, although the challenges or problems they face may be clear and often they are not, there tends to be disagreement about how to address them. And when they achieve a solid solution to a given problem, each situation is so dynamic that only rarely is the solution replicable in other times or contexts. In these complex situations faced by a Woord en Daad trajectory, the methodologies, models, or logical frameworks of results-based management rarely are useful. They assume that you can predict a series of cause/effect relationships in the logical chain inputs activities outputs outcomes impact when those relationships are fundamentally unpredictable. In contrast, a theory of change (ToC) presents a broad view of the desired change and the chosen strategy to contribute to that change. The ToC model that I propose, identifies the assumptions, strategies, actors and results at the level of outcomes that are expected to interact and enhance one another in the campaigning. In order to set some kind of baseline a specific Theory of Change will be re-constructed for each of the seven sample trajectories of the PI North. 3 The ToC for the ad-hoc lobby will be of a more generic nature. The theories of change will be reconstructed based upon written information provided by Woord en Daad and interviews with the staff. That staff will be Evert-Jan Brouwer plus those staff that is knowledgeable of and/or responsible for the issue of the trajectory. I assume that this will include 1 or 2 persons in addition to Evert-Jan Brouwer per issue, and that combinations of people will be possible/logical. 2 See the universal standards for evaluation of the American Evaluation Association at www.eval.org. 3 In other words, these will not represent the general theory of change of Woord en Daad as organization. 4
Each ToC will clarify intended outcomes; necessary preconditions to achieve these; alternative pathways (lobby and advocacy strategies and interventions); assumptions about behavioural changes of other actors and/or external contexts and planned contributions by the consortium to achieve intended outcomes. This is the ToC format that I use as starting point. Theory of change: Basic model MISSION What is the purpose, the reason for being of the programme? ASSUMPTIONS What is the logic that the programme uses to explain that if A, B and C happens it will result in X, Y and Z? RESULTS What outcomes does the programme aim to influence? STRATEGIES What and how will the programme do to contribute to the occurrence of A, B and C? TARGETED SOCIAL ACTORS Who does the programme intend to influence? To whom are its strategies targeted? Step 2: Outcome Harvesting In this evaluation, outcomes are understood as: Observable changes in the behaviour, relationships, activities, policies or practices of the people, groups, or organizations that have contributed, or potentially contribute to Woord en Daad s objectives. These changes in behaviour are beyond Woord en Daad s control but which they have influenced. These changes must be reasonably linked to one of the seven PI North sample trajectories, although they are not necessarily directly caused by them. In addition to identifying and formulating the outcome, I want to gather information that helps to understand its significance (in the light of W&D objectives) and the way in which Woord en Daad contributed to the change. Outcomes differ from outputs, which are immediate results of your work and which you fairly well control. That is, outputs are the processes, goods and services that you produce, such as publications, campaigns, and advocacy activities, to name a few. Outputs are very important and in no way are they inferior to outcomes; indeed, without outputs, few outcomes would be realized. They simply are a different type of result. In approaching this task, seeking linear, cause-effect relationships must be avoided between what Woord en Daad did and the changes it influenced in other social actors. Woord en Daad and certainly the environment in which it operates are highly complex, open and dynamic. Thus, causality tends to be messy, multi-level and multi-directional, as well as unpredictable. Thus, 5
some words of caution are in order: - I understand that you usually work in concert with others and rarely wholly alone and therefore frequently your activities will contribute indirectly and partially to one or more outcomes. - Outcomes often take time to emerge and some activities may never lead to one. - Outcomes may be generated intentionally but sometimes unintentionally, and often unexpectedly. - Outcomes are part of a process of change or a pathway of change, meaning that several outcomes can influence each other over time. The Theory of Change model and the outcome harvesting are designed to be able to see and discuss these processes or pathways. - Influencing a social actor not to take action avoiding something undesirable from happening can be a significant outcome. These outcomes are formulated as the social actor changing its expected negative behaviour, relationship, activity, policy or practice. - Outcomes may be positive or negative, which undermine rather than enhance your goals. We encourage you to share negative outcomes when the damage caused or what you learned was significant. The annex shows an example of an outcome description. The used format can and should be tailor-made for the evaluation of Woord en Daad. For example, in order to be able to answer evaluation question 4, a sub-section on the added value for the needs in the South should be added to the section on the significance of the outcome. The Outcome Harvesting approach has six steps. Here are the first three: - Design of the Outcome Harvest: Commissioner and evaluator agree what information is to be collected in addition to the change in the social actor. - Review of documentation and draft outcome descriptions: Extraction of outcomes by evaluators from reports, press releases and any other material Woord en Daad has on file documenting outcomes and activities to achieve them. - Engagement with Woord en Daad staff: Evaluator engages directly with the informants to review what was extracted from the files, identify and formulate additional outcomes and classify them all. Informants will often consult with others. After the first two steps, the ToC has been reconstructed and the achieved outcomes have been described. The source of information in these two steps is the staff from Woord en Daad and its documentation. Remind that this is an explicit choice as it has an added value to describe the theories of change and the achieved outcomes from the viewpoint of Woord en Daad itself. The next step will contextualize and substantiate this based on data from others who reflect on the info from Woord en Daad AND who will add their viewpoint. Together these will be analyzed 6
in step 4. Step 3: Substantiation of the outcomes through external interviews/survey In this step, the evaluator will obtain the views of independent people knowledgeable about the trajectories, its strategic choices and, most important, the outcomes. It is my experience that the outcome descriptions are a great basis for these interviews because the other knowledgeable people have very valuable information to reflect on. The evaluation questions 1, 2 and 5 will be included in these semi-structured interviews. Personal meetings for in-depth interviews are a great way to obtain the views of these external informants. But this is also resource-intensive. A survey also delivers valuable data, it is fast and cheaper. I therefore propose to do in-depth interviews with 1-2 externals per trajectory, plus minus 10 in total, and 15-20 interviews through a survey. In any case, this approach enhances the validity as well as the credibility of the findings. Step 3b: Other internal interviews In addition to the above, the evaluator will execute interviews with two members of the Executive Board of Woord en Daad. 4 This is to substantiate the ToC and the harvested outcomes and, more specific, to collect data for evaluation questions 3, 4 and 5. Step 4: Analysis and interpretation of the information to answer the evaluation questions The data collected in the previous three steps are sufficient to answer the five evaluation questions. Important elements of the analysis and interpretation are: - Categorization of the achieved outcomes according to a specific model of outcome indicators 5 - Reflection of achieved outcomes with the Theory of Change. This will include the pathways of change, the significance of the results, Woord en Daad s contribution and the opinion of the external informants - Effectiveness (question 5) will be assessed in the light of Woord en Daad s theory of change 4 Remind that information from Evert-Jan Brouwer and other knowledgeable and responsible policy staff is extensively harvested during steps 1 and 2. 5 I will apply a model that is being developed during the MFS II evaluation of International Lobbying & Advocacy 7
Step 5: Comments on draft and learning workshop After comments from Woord en Daad on the draft report, I propose to organize a learning workshop, which includes all relevant staff. Roles and responsibilities In order to avoid substantive, administration and communication problems I here delineate roles and responsibilities of the evaluator and of the Woord en Daad staff: Woord en Daad: - Report seven extended trajectory descriptions and nine limited trajectory report or structured sets of documentation before the evaluators starts the data collection - Respond to proposed methodological choices and together with evaluator finalize the detailed evaluation approach - Provide all necessary information to the evaluator and give access to documents - Relevant staff to join a session in which the Theory of Change is reconstructed - Relevant staff to be involved in the outcomes harvesting process - Deliver names and contact of external informants - Introduce the evaluator to the external informants - Respond to draft report - Organize learning workshop after final reporting - Wouter Rijneveld will manage the contract and serve as the evaluand s liaison with the evaluator Evaluator: - Collect and analyze all data, including meetings with Woord en Daad, external interviews and surveys - Present findings and write draft report - Adapt draft report after comments from Woord en Daad - Facilitate learning workshop based on evaluation findings - All raw data are kept confidential Reporting Woord en Daad will deliver seven sub-reports with each an extended description of one of the seven sample trajectories of the Northern Policy Influencing programme and nine short descriptions of other, less relevant, trajectories of the past. The evaluator will prepare a report in which the evaluative questions will be answered for the Policy Influencing North programme. Each evaluation question will be answered for each of te 7 trajectories plus for the program in general. The evaluation report will include 8
recommendations of questions or points for discussion. The evaluation report will be public and the evaluator has the permission to publish information from or based on the evaluation. Upon request the evaluator can return the completed data sets (filled out questionnaires, surveys, interview notes and tapes, etc.). 9
Annex Example outcome WRM (World Rainforest Movement) Outcomes formulated by: Ricardo Carrere (rcarrere@wrm.org.uy) Evaluator responsible: Wolfgang Richert Outcome: In 2006, the Eighth Conference of the Parties of the United Nations Convention on Biological Diversity recommended that countries exercise caution when approaching the potential use of genetically engineered trees. Description: In March 2006 the eighth meeting of the UN Convention on Biodiversity (CBD) meeting that took place in Curitiba, Brazil, 160 national governments took the following decision: The Conference of the Parties, recognizing the uncertainties related to the potential environmental and socio-economic impacts, including long-term and trans-boundary impacts, of genetically modified trees on global forest biological diversity, as well as on the livelihoods of indigenous and local communities, and given the absence of reliable data and of capacity in some countries to undertake risk assessments and to evaluate those potential impacts recommends parties to take a precautionary approach when addressing the issue of genetically modified trees. The decision also called for the initiation of a global compilation of data on the social and environmental implications of genetically engineered (GE) tree release, in a process that includes the participation of relevant organizations, including indigenous and local communities. Before the decision was adopted, the CBD had no position regarding GE Trees. Significance: GE Trees are being heavily promoted by forestry corporations as the modifications that are being investigated will result in the increase of corporations profits. However, GE trees are one of the most dangerous threats to forests, which host most of the Earth s terrestrial biodiversity. The release of GE trees will inevitably and irreversibly contaminate forest ecosystems and impact heavily on biodiversity. The COP-8 decision provided a political leverage point for organizations lobbying the CBD. Pressure to the CBD was raised again during COP 9 in May 2008, and a request for a formal moratorium to the release of GE trees was presented by many CSO. As a result, the CBD expanded the decision taken at COP 8. The CBD outcome, recommending a precautionary approach to GE trees, represents a first step in recognizing the dangers of GE trees. It did assist NGOs and scientists alike in sending an urgent alert to all nations that there is insufficient scientific data on the implications of GE trees, which pose a threat to forests and indigenous and local peoples globally - and therefore it is crucial to halt all releases at least until sufficient data and assessments become available. As an example of the importance of this outcome and how NGOs used this decision, during COP 9 (May 2008) the issue was hotly debated and African countries and a number of countries in Latin America unified in their support for stopping the release of GE trees. 10
Contribution of the counterpart: WRM is the Southern Hub of the STOP GE Trees Campaign, and has been campaigning against the release of GE trees for a long period. As an example of our long involvement in the issue, one of the first critical books on GE trees was published by WRM in 2004. In 2006, together with Global Justice Ecology Project, STOP GE Trees Campaign, EcoNexus, Friends of the Earth International and Global Forest Coalition WRM organized a campaign to influence the CBD COP 8. Activities ranged from press releases delivered to the international media to specific letters addressed to the CBD representatives, signed on by hundreds of people from many countries world wide. A statement by the Latin American Network against Tree Plantations was presented also to the COP in Curitiba (WRM facilitates this network). A special bulletin focused on Biodiversity was released previous to the COP meeting. 11