Institutionalizing and Sustaining the Lot Quality Assurance Methodology in Uganda:

Transcription

1 STAR-E LQAS Institutionalizing and Sustaining the Lot Quality Assurance Methodology in Uganda: How close are we? June 26, 2014 Jerald Hage Director, Center for Innovation University of Maryland College Park, MD Joseph Valadez Professor of International Health Liverpool School of Tropical Medicine Liverpool, UK With Charles Nkolo Liverpool School of Tropical Medicine

2 Copyright Liverpool School of Tropical Medicine 2014 All rights are reserved. This report and any attachments to it may be confidential and are intended solely for the use of the organisation to whom it is addressed. No part of this report may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photo- copying, recording or otherwise without the permission of Liverpool School of Tropical Medicine. The information contained in this report is believed to be accurate at the time of production. Whilst every care has been taken to ensure that the information is accurate, Liverpool School of Tropical Medicine can accept no responsibility, legal or otherwise, for any errors or omissions or for changes to details given to the text or sponsored material. The views expressed in this report are not necessarily those of Liverpool School of Tropical Medicine.

3 TABLE OF CONTENTS Acronyms... 4 Acknowledgements... 5 Executive Summary... 6 Background... 8 Research Methodology Measurement Section Measuring Institutionalization Measuring Challenges Measuring Learning Measuring Sustainability and Challenges Research Findings The Degree of Institutionalization Challenges Learning and Challenges Sustainability and Challenges Recommendations Short- Term Recommendations Long- term Recommendations Appendices Appendix A: Discussion of the Research Design Appendix B: DHO and Assistant DHO Interview Questionnaires, Focus Group Schedules, and Surveys for Supervisors and Data Collectors Appendix C Monitoring Report on LQAS/HFA survey data use by districts References:... 57

4 ACRONYMS CAO CDC CSF CSO DHO GoU HF HMIS INGO LSTM LQAS MCH MoH MoLG MSH NGO NLF OVC PACE SDS TASO TWG VHT USAID Chief Administrative Officer Center for Disease Control Civil Society Fund Civil society organizations District Health Officer Government of Uganda Health Facilities Health Management Information System International non- government organizations Liverpool School of Tropical Medicine Lot Quality Assurance Sampling Mother and child health Ministry of Health Ministry of Local Government Management Sciences for Health Non- government organizations National LQAS Facilitators Orphans and Vulnerable Children Program for Accessible Health Communication Strengthening Decentralization for Sustainability The Aids Support Organization Technical Working Group Village Health Teams United States Agency for International Development

5 ACKNOWLEDGEMENTS LSTM would like to thank all of the many stakeholders from the nine districts In particular they would like to single out Esther Sempiira, the national coordinator of STAR- E LQAS and Stephen Lwanga, MSH Country Representative and Director of Management Sciences for Health, Uganda. Special thanks also goes to the 7 District Health Officers and 6 Assistant DHOs or their substitutes and the 69 supervisors and data collectors of all nine participating districts who took the time to answer the questions and to discuss possible strategies of sustainability.

6 EXECUTIVE SUMMARY The Liverpool School of Tropical Medicine requested that a research study of the extent of institutionalization, the challenges remaining, and the potential for sustainability be made of the use of Lot Quality Assurance Sampling (LQAS) in Uganda since 2009 with funding by USAID called STAR- E LQAS. In addition, the Management Sciences for Health (MSH) asked that the study also examine how LQAS has impacted on the effectiveness in the delivery of social services. In particular, they desired recommendations that could be implemented in the remaining nine months of the current contract so as to improve service delivery. With these instructions, a research study of nine districts Bushenyi, Mbarara, Kabale, Kabarole, Hoima, Kamwenge, Mbale, Kaberamaido, Tororo- - within Uganda was undertaken during the month of May The nine districts were selected to maximize variation in the percentage of births delivered in a health facility within three regions: eastern, western, and southwestern. The reasons for the specific research design, its advantages and disadvantages are explained in the research methods section and Appendix A. The main reason for a comparative case method research design is to be able to relate the use of LQAS to the effectiveness of maternal health. Interviews were conducted with 7 District Health Officers (DHO) and 6 Assistant DHOs to cross- check on institutionalization, challenges, learning and sustainability. Similarly 31 supervisors and 38 data collectors participated in separate focus groups for the same reason as well as filled out surveys. Institutionalization was measured by two indices each composed of four components: (1) the degree of training and experience; and (2) the degree of coordination and control of the LQAS process. On the first index, five districts scored 75% or higher and on the second, all but two districts scored this high. The second index is the more critical of the two because it measures how much the results of LQAS are employed in planning and budgeting. The second index demonstrates that for coordination and control more institutionalization has occurred than with training and experience, indicating that the STAR- E LQAS intervention has been particularly successful in integrating monitoring and evaluation into planning and in precipitating intervention strategies. However, a contrast between Hoima and Kabermaido indicates that it is preferable to support at minimum two rounds of LQAS data collection. In Uganda 13 districts have had only one round and 32 districts have not yet been introduced to LQAS. To answer the question of whether LQAS has had an impact on the effectiveness of service delivery the two indices of institutionalization were added together and their rank order correlated with the rank order of the percent of mothers having deliveries in a health facility; the Spearman Rho is.45. To substantiate the causal connection, a number of examples were taken from the interviews, especially with the DHO and the Assistant DHO (MCH) or their substitutes (see Table A.2), on how LQAS was used to improve social services. In addition, an index of availability of health facilities (HF) was constructed on the basis of the number of HF s per 1000 population located in a 100 square kilometre catchment area. The Spearman Rho correlation between this measure and delivery of births in HFs is.75.

7 However, Bushenyi, which cored 100% on both indices or high institutionalization; while it has a relative low number of HFs it does have a very high percentage of births in HFs. Thus, it represents a model for how LQAS can be used to improve social services. The usual challenges are the problems of transportation, lack of rain gear, insufficient budgeting for translations, and allowances for over night stays in parishes and sub- districts that are difficult to reach during the process of data collection. The more unusual challenge is the need to continually train new appointees to the positions of DHO and Assistant DHO. Those with no exposure have little interest in this methodology. Learning was measured in multiple ways. LQAS provides a major mechanism for learning about sub- districts that are not performing well and adjusting strategies to improve service delivery. But the DHOs and the Assistant DHOS (MCH) reported learning in other ways as well, especially about maternal and child health. Thus, both learning mechanisms account for the improvement in services. With the available data is not possible to estimate the relative importance of either mechanism. Considerable support exists for two strategies of sustainability from the health management team, the supervisors, and the data collectors. The first strategy is to integrate the data collection into the routine work of the supervisors and the data collectors. In the process some of the challenges in data collection can be solved including allowing more time to collect the data and better scheduling, providing better transportation, and conducting more translations. The second strategy is to transfer the knowledge about LQAS from the districts that scored high on institutionalization to those that scored low or have no exposure. The National LQAS Facilitators (NLFs) represent an elite corps for this mission. Beyond this, the support for LQAS on the part of the health management team is extremely strong, when they have received reports and have been involved in the process. The short- term recommendations for the next nine months are: 1. Attempt to include members of other ministries in the Technical Working Group being formed by Patrick Mutabwire, Director of Local Government Administration. 2. Support a second round of LQAS data collection in the 13 districts that so far have had only one round. The long- term recommendations for the period afterwards are: 1. Hold planning committees among districts with similar sets of challenges to discuss how best to integrate the data collection process in the on- going work of the supervisors and data collectors in those districts to improve quality of data collection, lower stress and develop creative solutions to various challenges. 2. Begin to train new data collectors and supervisors in those districts that as yet have not had any exposure to LQAS. 3. Every several years train the newly appointed DHOs and Assistant DHOs (MCH) in the LQAS process so that the health management team becomes committed to this process and uses the data in their planning. If resources permit include other district managers to build support for LQAS.

8 BACKGROUND USAID has been supporting Uganda since 2009, to aid districts to use Lot Quality Assurance Sampling (LQAS) to manage and improve their health programs. This work is undertaken by a Management Sciences for Health (MSH) managed project called STAR- E. The LQAS component of the STAR- E project, STAR- E LQAS, is implemented jointly with the Liverpool School of Tropical Medicine (LSTM). Since the time they first started working in 10 districts during 2009, now close to 80% of the 112 districts have used LQAS at least once. Given this effort, USAID and MSH have been interested in assessing what has been accomplished, and the extent to which LQAS has been accepted in Uganda. To make this evaluation, we asked the following three questions: (1) How much has the LQAS methodology been institutionalized? (2) What challenges remain to increase the degree of institutionalization? (3) What are some pragmatic strategies for ensuring sustainability of this methodology? The answers to questions 2 and 3 can provide a road map for the remaining nine months of the current project extension, and also advise USAID and the Government of Uganda (GoU) about future next steps in supporting LQAS work. The definition of institutionalization that guides this research project is: Institutions are clusters of norms with strong but variable mechanisms of support and enforcement that regulate and sustain an important area of social life. Specifically, norms are nothing more than the rules of the game, e.g. how often is LQAS conducted and what procedures are employed for the presentation of results. The most important clusters of norms are jobs or who does what, rules about supervision and enforcement of various regulations about what should be done. In addition to these questions, we considered another three queries that are important for the country of Uganda. LQAS is a means to an end, namely, the improvement in social services for the targeted populations, whether they be men with HIV, pregnant mothers, young children and their mothers, or orphaned and vulnerable children (OVC). It is important to keep a clear distinction between the effectiveness of LQAS data collection procedures and their impact on the quantity and quality of social services. We can examine that issue by observing how much the data are actually used in planning and budgeting for social services to help the people of Uganda. This distinction is an important one. The reason why is that one wants to avoid the error of Dr. Freud who thought awareness is enough and individuals will change their behavior once they understand their problems. As 8

9 is well known, this is not always true and one might add it is even more difficult for organizations, versus individuals, to change their programs, policies, and business- as- usual. Indicating poor performance does not necessarily produce changes in organizational strategies. Part of the reason may simply be because a root cause is not easily changeable, e.g. the absence of health care facilities. In these circumstances, LQAS should at least be given credit for the recognition of the problem. The opposite is also the case. Sometimes improvements are due to reasons other than the use of LQAS. One needs to avoid assigning credit when the improvement in effectiveness is a consequence of the intervention of some external agent, such as the Baylor Medical School (working in Uganda) or a training project of the central government. Therefore, identifying when LQAS has had an impact and when it has not is an important component of this evaluation. Thus, the parallel set of questions for the benefit of the Ugandan people are: (1) What impact has LQAS had on the effectiveness in the delivery of social services? (2) What challenges remain to improve the impact of LQAS on service delivery? (3) What are some pragmatic strategies for increasing the impact of LQAS on service delivery? In answering these questions, it is important to understand the historical experience of Uganda with LQAS. Prior to STAR- E, MSH has had a long history for supporting health services in Uganda while Prof. Joe Valadez (currently in LSTM) had previously introduced this methodology (2003) into 30 districts through the Uganda AIDS Council for assessing AIDS/HIV prevention and control through the World Bank. In a commendable and prescient innovation encouraged by USAID, eight major USAID projects (STAR- E, STAR- EC, STAR- SW, STRIDES, NU- HITES, SUNRISE, CSF and SMP), despite their quite disparate objectives agreed to use the same set of 59 indicators to assess: (1) reproductive health and family planning; (2) child health; (3) malaria; (4) sexually transmitted diseases including HIV counseling and prevention; (5) tuberculosis; (6) water and sanitation; (7) nutrition; (8) orphans and other vulnerable children; and (9) education. The seven projects involved four distinctive implementing partners: MSH, JSI Research and Training, Inc., Elizabeth Glaser Pediatric AIDS Foundation, Plan International, Inc. and Johns Hopkins University. MSH/LSTM partnership provided technical assistance in the use of the LQAS methodology to these partners. In addition to these projects, USAID has funded the SUNRISE Project to help OVC left behind because their parents have died of AIDS, and Strengthening Decentralization for Sustainability (SDS) Project, another HIV/AIDS project currently working in 35 districts. In addition, many international non- government organizations (INGOs) work in tandem with local non- government organizations (NGOs); these organizations include AfriCare, AidChild, and the Program for Accessible Health Communications and Education (PACE), and The AIDS Support Organization (TASO). Further LQAS support has come from UNICEF who is working with Child Fund International and LSTM to use LQAS together with other management methods to support decision- making for Integrated Community Case Management of child health (21 Districts); and lastly, UKAID through the Department for International 9

10 Development (DfID) support use of LQAS in the Acholi and Lango Regions to assess the effectiveness of Results Based Financing. At various times since 2009, other international organizations have had an impact on the effectiveness of the delivery of services such as the Baylor College of Medicine with support of Centers for Disease Control (CDC) in the U.S. and its program for training village health team members (VHTs) that proved highly effective in the districts in which they operated. Another important international consortium is the Civil Society Fund (CSF), which receives funding from six countries. This fund is primarily engaged in various projects designed to make local institutions stronger relative to HIV/AIDS prevention and helping OVC. This Fund asks for proposals from civil society organizations (CSOs) and once funded one of the nine districts selected for this study. Finally, the government of Uganda also has introduced new policies that impacted on the delivery of services. Thus, this combination of interventions funded by USAID, other international agencies, and the Uganda government is quite complex. Due to the complexities concerning the number of actors involved in this work, this assessment will have a challenge to sort out the various intervening factors. Nevertheless, as the focus of this work is to understand the institutionalization of LQAS in Uganda (rather than in attributing institutionalization to any particular donor), our principal responsibility here is to assessing the sustainability of LQAS in Uganda. The report has four sections. The first describes the research methodology while the second focuses on the critical problem of how to measure institutionalization and sustainability, hardly easy tasks. Given space limitation, additional detail is provided in Appendices A and B respectively. Section three contains the research findings about how much institutionalization has occurred and strategies for sustainability followed by the recommendations in the fourth section. Throughout, this assessment has been designed to represent a model for what USAID can accomplish in other evaluations, and in other countries by supporting other countries to institutionalize LQAS. In particular, it demonstrates the advantages of studying institutionalization and sustainability at the district level. 10

11 RESEARCH METHODOLOGY The research strategy for answering these six questions is to focus at the district level and their management by local government authorities in contrast to much of the literature that discusses institutionalization and sustainability at the central government level (Max de Xaxas and Vogel, 2007; Stash et al., 2012; USAID, 2012). The projects unfolded at the district level and therefore, it is only at this level can these questions be adequately answered. Furthermore, this emphasis is consistent with the GoU s devolution of authority to the districts in the 1990s. Given the concerns about ownership and social capital in the international literature, the district level appears to us to be the more appropriate level for measuring these concepts. Another important reason to work at the district level is the considerable variation in the number of times that LQAS has been used to collect data at this level, with some districts having experienced this process five or six times (Bushenyi, Kabale) and some only once (Hoima and Kaberamaido) (See Table A. 1, for a complete listing of all districts and years in which data was collected before 2013). Certainly understanding challenges to institutionalization requires interviewing people directly about their experiences. Some districts are largely rural and others urban; some districts have fairly adequate coverage with health facilities (HF) (e.g. the south- western region), and others do not (e.g., northwest). Some districts have VHT or village health team members (hereafter VHTs) with better basic education than do others. When selected, the VHTs need to have some customized training and this is in many cases donor supported, e.g., by the Baylor College of Medicine and U.S. CDC among others. Both the availability of HFs and the quality of training of VHTs are highlighted because of their impact on the effectiveness in the delivery of health services quite independently of LQAS. However, the most important reason for studying the district level is to determine whether various strategies for sustainability are viable and what challenges have to be overcome to sustain this process of monitoring and evaluation. Again, only those people and institutions directly involved in LQAS can make these assessments. To capture different degrees of institutionalization and potentials for sustainability at the district level, the research design included nine districts that varied in mothers accessing maternal health care services; specifically, we assessed three districts in each of three regions. The selection of districts (see Table 1) is designed to maximize the variation within region. This is a critical point because the conventional wisdom is that there are significant differences between the regions. If so, then the question becomes how well do districts facing approximately the same conditions as reflected in regional differences utilize LQAS in their planning and choice of interventions. We operationalize this selection by using one indicator, namely, district effectiveness in women delivering their babies in a health facility; 11

12 2012 was used as the index year as all districts had data for that year. These data were obtained directly from the STAR- E LQAS super- dataset that includes all LQAS data from Table 1 Percent of Mothers Who Delivered Their Child in a Health Facility in 2012 District Region Percent 2012 Other LQAS Surveys Also Measured In 2013 Bushenyi Southwest , 2010, 2006, 2003 Yes Mbarara Southwest , 2003 No Kabale Southwest , 2010, 2006, 2003 Yes Kabarole Western , 2004 No Hoima Western No Kamwenge Western Yes Mbale Eastern , 2010, 2006, 2003 Yes Kaberamaido Eastern No Tororo Eastern , 2004 No The nature of this research design, a comparative case study, has both advantages and disadvantages. It is perhaps ironic to evaluate a method such as LQAS that places a premium on random selection with a design where districts have been chosen purposively. The justification for this procedure is the need not only to answer the first three questions indicated above specifically related to LQAS, which could be examined with a stratified random design of districts classified by region and number of LQAS visits. The justification of the purposive sample was to connect the second set of questions, impact on health services, with the use of LQAS. It is for this reason that this comparative case method design was utilized and in the jargon of social research, sampling occurred by using the dependent variable. This design is based on the assumption that potentially different levels of LQAS institutionalization can explain variation in percent of births in hospitals reported in Table 1. The data collection for this study consists of two interviews and two focus groups at the district level. A basic principle was to attempt to obtain information from two different sources for each major concept. For example, the DHO was interviewed about learning relative to two separate indicators of safe motherhood (antenatal visits and delivery in a hospital) while the Deputy DHO was asked primarily about learning relative to two indicators of child health (a mother s treatment seeking behavior for her sick child and immunization). In some districts, the DHO or the Deputy was absent and in others they were too busy preparing reports, participating in meetings or absent on trips. Wherever possible we tried to conduct telephone interviews with the managers who were not present on the day of our visit. Also, at times we selected knowledgeable substitutes (health educators) to ask the same battery of questions (see Table A.2). Seven DHOs and six Assistant DHOs participated 12

13 in the study. In addition, wherever possible, documentation of the impact of LQAS on planning was secured to crosscheck and supplement the responses received in the interviews. Separately 31 supervisors and 38 data collectors (see Table A.2) participated in focus groups about various challenges and strategies for sustainability. Again, both were asked many of the same questions to obtain a crosscheck on the information. The use of focus groups was necessary because there was not enough time to interview three to five members in each group separately. Individual data was obtained by asking each supervisor and each data collector to fill out a short survey about their position within health districts, their responsibilities, and their level of LQAS training. The classifications of their major work responsibilities are contained in Table A.3. As can be observed, the bulk of the data collectors and the supervisors worked either in the health sector (43%) or in community development (33%). An extended discussion of the research design and why specific regions were selected is contained in Appendix A. Initially the research design included the idea of contrasting villages by the extent of the cooperation among community members (or, in other words, by the presence of social capital) since this is a hidden resource and given its importance in the international development literature (Mas de Xaxas and Vogel, 2007). But as the focus groups unfolded it became clear that there was not much variation between sub- districts or parishes in the case of Uganda. But since this study is concerned with a general framework, it is important to consider the importance of sub- unit variation even if not relevant in this country; in other countries we have found social capital to be an important factor (Valadez, Vargas and Hage; 2005). Measurement Section Projects have limited time periods. In the case of Uganda, most of the USAID projects were initially funded for a period of five years, and presently some of them have been extended for one year. The issue then arises if USAID funding ceases, whether data could continue to be collected and used in the normal planning process. LQAS is a complex set of activities, with different individuals performing distinctive duties; this complexity is one reason why both interviews and focus groups were used to collect the data as we needed to assess individual functions and then the different functions working together. Since institutionalization and sustainability are so critical to USAID and other international funding agencies, considerable time is spent in this section on how to operationalize these concepts and measure them, especially at the district level. In addition these concepts are complex and difficult to measure (see Hage, 1972 for strategies of turning constructs into operational indicators and indices). Institutionalization is measured in two different ways: 13

14 (1) training and experience of local staff; and (2) more critically, their coordination and social control (or regulation) (see Figure B.1). The problem of challenges is explored in several different ways and at the level of data collection (see Figure B.2) as well as in the impact of LQAS on the effectiveness of services (Figure B.3). To allow for other reasons for improvement in the effectiveness of health services, both the DHO and the Assistant DHO were asked how they learned about new strategies of interventions relative to promoting both maternal and child health respectively (see Figure B.4). Finally, three strategies for sustainability were explored in both the interviews and the focus groups (Figure B.5). Measuring Institutionalization The concrete questions that we asked about the level of training and experience of the data collectors and of the supervisors are indicated in Figure B.1. The central reasoning is that as the number of times the individuals within a specific district have participated in LQAS training and in the collection of data increases, the more likely they are to be competent and capable of continuing if project funding is no longer available. In addition, the supervisors were asked to evaluate the competence of their data collectors. Following this question, we asked the supervisors if they felt that the data collectors were capable of collecting data without supervision. For the supervisors, we asked what their duties are as a check on their knowledge level. In addition to the supervisors, there is a distinctive category called the National LQAS Facilitator (NLF). No separate interview was developed for this category but several were involved in the focus groups for the supervisors. Both the DHO and the Assistant DHO were also asked about their exposure to LQAS and whether they had participated. Some reported that they had been supervisors at one point in time. This turned out to be far more important for the impact of LQAS on the effectiveness of social services than the training and experience of the data collectors and supervisors. As is indicated in the discussion of challenges, when the DHOs and the Assistant DHOs have not been exposed to LQAS or instructed about it, they have little commitment to it. This is an important point as it indicates that the comments we obtained from active local authorities has validity, and they were not responding to some assumed preferred response. At the level of the district two indices were constructed from the various indicators reported in Figure B.1. The first index, the degree of training and experience, which is reported in Figure One, has a maximum score of 12 and is based on four indicators each one being scored 0-3. For each year that the DHO, Assistant DHO, supervisors and staff who participated in data collection one point is assigned up to a maximum of three. For the two levels of district health management, that is the DHO and Assistant DHO, the key issue was did they receive reports and not whether they had been trained in LQAS data collection 14

15 procedures. If one or another of these levels of management or their substitutes was missing or if they were new and had not received any reports, a zero was assigned because they had had no exposure to LQAS. This was a problem in several districts (see Table A.2). At the level of the supervisors and the data collectors means were constructed but in most cases, there was little variation because most of the supervisors and data collectors had participated in all rounds of data collection in that district since A few individuals had also been involved in the earlier rounds mentioned in Table 1. Institutionalization is not only a question of training and experience. Perhaps the more important ingredient is whether coordination and social control exist over the activities or duties of the data collectors. A process such as LQAS must not only be coordinated, different people doing disparate acts at specific times, but the actions of the individuals performing the disparate duties must be controlled, providing some assurance that they perform their responsibilities competently. Since coordination and control can be accomplished via a number of mechanisms (see Hage, 1974), three specific ones were measured and at multiple levels. Unlike the previous index that counts frequencies, this second index includes a set of indicators concerning the last time LQAS data were collected, whether this was in 2013 or In the latter instance there were some recall problems in one district, especially among the data collectors. Wherever possible the answers were checked against reports to be certain that an accurate assessment was being made. Again, the index has four quantitative components: (1) use of manuals; (2) quality of data collection control; (3) report of the use of LQAS to change strategies by the data collectors and by the supervisors; and (4) the report of the use of LQAS results in planning and budgeting by the DHOs and the Assistant DHOs. One point is given if the supervisors report the presence of a manual for supervision and one point if the data collectors also report they had a manual. Two points were assigned for two different ways of checking on the quality of the data by the supervisors. Component 3 was worth four points while component 4 was worth another four points. The most important components in coordination are whether LQAS results have been used to change health strategies, and in planning and budgeting them. The answers to questions about this begin to lay the groundwork for understanding whether the LQAS monitoring and evaluation system is leading to changes in the effectiveness of the delivery of social services. The focus groups focused on the third component while direct questions were asked of both the DHO and the Assistant DHO but relative to different areas, in the former instance safe motherhood, and in the latter child health. In the context of the interviews, DHOs and the Assistant DHOs volunteered other areas where the LQAS results were incorporated into planning. 15

16 Measuring Challenges A second objective of this research study, in the first three questions listed above, is the measurement of challenges to the institutionalization of LQAS. Several distinctive kinds of challenges were explored in the focus groups and the interviews with the DHO and the Assistant DHO (see question 2 for each group listed in Figure B.2). The supervisors and data collectors were asked about challenges that they had in collecting the data. In contrast, the DHOs and the Assistant DHOs were asked about how LQAS could be made more effective. In addition, we enquired about the normal work of the data collectors and the supervisors in the surveys handed out after the focus group session was over (see Table A.3). This information is critical because in fact most of them are employed full time. One can imagine that there can be a conflict between their normal work and the special task of conducting LQAS community surveys. This issue is also relevant for evaluating the efficacy of one of the strategies for continued to collect LQAS survey data after donor agencies withdraw their support. A second and less obvious issue is whether collecting data across 59 indicators posed any special challenges. This was a concern that we explored on the recommendation of Esther Sempira, the LQAS national coordinator in STAR- E LQAS (see question 3 for the data collectors and the supervisors in Figure B.3). Finally, it is worth determining challenges represented by the use of the LQAS data by various government ministries. Uganda presents a certain paradox. Since 1996 Uganda has been decentralizing decision making to districts who have become key actors in most areas of government and especially in social services. Some projects such as the Civil Society Fund (CSF), as we have seen, are attempting to further this objective. But as decentralization has become highly institutionalized now after almost twenty years, it has made it more difficult for the two levels of government to interface. Therefore, an important set of questions explores whether the LQAS data at the district level is utilized by the central government (see questions 3 and 4 asked of the DHOs and the Assistant DHOs in Figure B.2) To measure this kind of challenge, the DHOs and the Assistant DHOs were asked in their specific areas whether or not their data had been transmitted to the central government and more importantly if it had been utilized. Measuring Learning While institutionalization and sustainability dominate the discussion of all donors working in developing countries, surprisingly little has been said about learning. Yet, learning is the essential reason for why LQAS is instituted. The assumption is made that once poor 16

17 performance is indicated, individuals will change their strategies and adopt more effective interventions. This requires learning and therefore learning should be included among the major concepts involved in monitoring and evaluation, that is asking how much has occurred. It is also a major paradigm within the organizational literature, where many useful ideas can be borrowed (see Argote and Miron- Spector, 2011). Two alternative explanations for how the districts learned and therefore why they varied in the effectiveness of having mothers deliver their babies in a health care facilitating were explored. The first hypothesis is that it is LQAS that makes the district aware of the problem and because of this information, new intervention strategies are developed and implemented by the implementing partner in cooperation with the local district (see question three asked in the focus groups with the data collectors and the supervisors as a cross- check, Figure B.3). Both the DHOs and the Assistant DHOs were asked if they used LQAS results in their planning. The second hypothesis is that independently of LQAS, the districts are learning and improving their strategies and tactics of intervention (see questions 2 and 4 for the DHO and the Assistant DHO in Figure B.4). Of course, it can be a combination of both of these. A great advantage of studying the institutionalization process of LQAS at the district level is the ability to examine whether there is a direct link between the presentation of results, the making of recommendations to correct particular deficiencies- - the red flags- - and their adoption. This is the central argument as to why LQAS should be used. Another advantage of asking these questions about presentations, recommendations and changes in strategies in various areas of health care is that it is a further check on whether the data collectors and especially the supervisors were following their instructions on how to use LQAS. The answers provide a cross- validation of the information obtained relative to the institutionalization of LQAS and especially for planning. From these questions several different kinds of narrative were developed. One of the most important ways in which learning was encouraged in the case of Uganda was the presentation of what are labelled preliminary reports to all the important stakeholders in the district including political, religious, donor agencies, the media and parish chiefs. At this moment, many of these stakeholders discovered problems, the red flags, that they had not realized existed As can be seen in the questions posed to the focus groups for the data collectors and the supervisors, it was left open as to which recommendations in any of the nine areas were made (see 2 asked in the focus groups with the data collectors and the supervisors, Figure B.3). In contrast, and consistent with the choice of safe motherhood and child health as major concerns, the interviews with the DHO and the Assistant DHO, the questions focused entirely on these two areas (see Figure B.4). 17

18 Establishing the link between LQAS and changes in the effectiveness of social services is crucial for the purposes of demonstrating the utility of monitoring and evaluation. Obviously this link cannot be studied at the central government level; it requires visiting a number of districts and asking concrete questions at multiple levels. But health districts, like any organization, can change their intervention strategies for many reasons other than simply monitoring and evaluation (see questions 2 and 4 of the schedule for DHOs and the Assistant DHOs in Figure B.4). It is important that any evaluation of LQAS be open to this alternative perspective in measuring institutionalization and sustainability and for multiple reasons. Perhaps the most important one is that if a district shows the capacity to continue learning even if the source of the learning is not from LQAS, it means that the district is more open to change and adaptation and fundamentally, this is what is most important for Uganda. It means that the district or at minimum its leadership is engaged in a problem solving mode and attempting to overcome obstacles. This is exactly what Uganda needs to make progress, not just in health care but also in other social service areas. Selecting DHOs or Assistant DHOs that have this capacity is perhaps the best strategy for diffusing best practices from one district to another including the use LQAS in planning. When health management is problem solving to improve both maternal and child health, it implies that they are much more open to change. As can be seen in Figure B.4, to measure this problem solving, management was asked about what were the initial intervention strategies for two indicators of safe motherhood and two indicators of child health and then how these strategies changed over time. These four questions- - when we could interview both of the DHO and the Assistant DHO or their substitutes- - provide a series of narratives in which we discovered a number of new aspects about the delivery of social services in Uganda. What emerged in several narratives was the causal agent of change were training interventions by either the Ministry of Health or a specific implementing partner, such as the Civil Society Fund, or by another international agency than USAID. These narratives became a learning device for this interviewer as well as a measure of the amount of learning that had occurred in these disparate districts and also a deeper understanding of the obstacles that management faces as they try to do their jobs. But implicitly they are also measuring whether management approaches their work as problem solving, recognizing that obstacles exist that prevent effectiveness, and attempt to overcome these obstacles. Another important source of learning and one that reflects a potential strategy for sustainability is the sharing of experiences across district levels (see questions 5 asked of the DHOs and Assistant DHOs in Figure B.5 and question 1 in this same figures used in the focus groups with the supervisors and data collectors). Both the supervisors and the data 18

19 collectors were asked if they ever participated in meetings with their counterparts in other districts. From these meetings can emerge what are called best practices, and individuals in one district can learn how to solve problems that they face with the experiences of those in other districts who have solved the same problem if these meetings are structured properly. Perhaps even more noteworthy is that this practice provides a mechanism for building morale and making the LQAS data collection process more meaningful to those individuals involved. The same questions were asked the DHOs and the Deputy DHOs. In addition to asking questions about participating in meetings about sharing, we interviewed Alice Nakagwa about whether STAR- E LQAS had organized meetings for this purpose and we received reports on three regional meetings in the Eastern District, Western and Central areas. Unfortunately, very few of the respondents that we interviewed or who participated in the focus groups had attended any of these meetings. Measuring Sustainability and Challenges The problem of sustainability has been a major issue in most development projects. Although a set of skills may have been institutionalized, the issue remains whether these skills will continue to be utilized. To answer the third question posed in the mandate that was given by the Liverpool School of Tropical Medicine, the interviews and the focus groups, contain several questions about alternative strategies that would ensure that the LQAS skills that had been acquired would continue to be used at a relatively low cost. The latter is an important constraint. Governments such as Uganda have limited resources and therefore, sustainability becomes the search for strategies that are of low cost. As can be observed in Figure B.5, the project explored three strategies. The first and most obvious one is: Can the data collection and analysis be integrated into the regular work routines of the data collectors and supervisors? In the beginning of the project STAR- E had made the critical decision to recruit individuals who would be trained from district level staff. As a consequence, the investment in human capital remains until these individual retire. An assessment of the possibility of integrating LQAS into the routine work of the data collectors and of the supervisors was asked at all four levels of the district assessment. Another strategy is to transfer knowledge from expert data collectors and supervisors to individuals in other districts that have either not received enough experience or have never been exposed to this method of monitoring and evaluation. Therefore, in the focus groups both the data collectors and the supervisors were asked if they could train others. 19

20 Finally, a third strategy reflects a point when LQAS is so successful that it is no longer necessary to collect this information. Instead the data analysis can rely on the recurrent Health Management Information System (HMIS). The reasoning is that if the percentage becomes close to 100 percent, perhaps it is no longer necessary to measure a specific indicator with a population- based survey. Only questions were asked about health services but the same logic could be applied to other kinds of social services, e.g. schools measuring attendance. These three strategies only concern the question of sustainability at the district level and not at the level of the central government. The project did not have enough time to explore fully this latter issue but did conduct several interviews about efforts at the central government that indicate some promising developments. 20

21 RESEARCH FINDINGS The Degree of Institutionalization The degree of institutionalization of LQAS is not a simple presence or absence; it comes in degrees. By measuring institutionalization in this way, it becomes much easier to understand how it might be further increased. Furthermore, the extent of institutionalization varies at two distinct levels of analysis: (1) in districts; and (2) in the central government. The data in this report primarily concerns the degree of institutionalization at the district level but some discussion is added about the central level, which is more than simply the sum of the 112 districts and their levels of institutionalization. As is indicated in Figure One, a bar chart, five of the districts have a score of 75% or better on the first index of institutionalization, namely training and experience. Three districts- - Bushenyi, Kabale, and Kamwenge- - scored 100% on this index. Admittedly, the major determinant of their score is how many times has the district participated in LQAS data collection but this is also the logic of how one counts experience. Those districts that had four or more LQAS rounds did not receive extra credit since it would appear three times is sufficient to say the skill levels involved in LQAS have been institutionalized. It might also be noted that consistently these individuals who had multiple rounds of experience also reported that they received refresher training. So no additional weight was given to the refresher as it was so closely associated with having participated in multiple LQAS rounds. The second index of institutionalization focuses on the degree of coordination and social control, which is probably of most concern to those interested in whether LQAS has been institutionalized. As can be observed in Figure Two, all but two districts- - Hoima and Kameramaido- - scored above 75% on this index. In other words, there is more institutionalization of coordination and regulating of the LQAS process (i.e., control) than there is training and experience. This is an important finding because it means that on the more critical index, district managers are incorporating LQAS results into the planning process; these nine districts at least have higher scores of institutionalization. On this index, two districts- - Bushenyi and Kabale- - scored 100%. Some examples of how the districts used the findings in the preliminary reports for planning and budgeting follow. The respondents in Kamwenge stated that they had used the results to secure additional funding and mentioned in particular attempts to increase their rates of immunization and male circumcision. Bushenyi was concerned by their low rate for having four or more antenatal care visits, and started working with the VHTs to get mothers to visit 21

22 the clinics. The VHTs were instructed to visit houses to emphasize exclusive breastfeeding. Also they trained the VHTs to train mothers in danger signs during their antenatal visits and political leaders were used to promote immunization. Finally, health management teams were very explicit about planning to build a new health facility each year and training the staff for it on the basis of the LQAS results. Mbale also used the LQAS results to plan infra- structure. Following many of the same procedures that Bushenyi did, in Mbale the VHTs escorted mothers to the HF. They also encouraged fathers to transport the mothers. In sum, the DHO in Mbale felt that LQAS pushed them to take more responsibilities for their district. Figure One Degree of Institutionalization Based on Training and Experience: Percentages Computed on a Total of 12 points 22

23 Figure Two Degree of Institutionalization Based on Coordination and Control: Percentages Computed on a Total of 12 Although Kaberamaido scored relatively low in the degree of institutionalization, they did report the use of LQAS data to indicate that there was a basic problem with the level of latrines in some of the sub- districts and this issue was addressed. The scores on breastfeeding were used in Tororo to inform changes in their interventions. But this district also illustrates a challenge when the LQAS produces too much demand for HF, resulting in long lines at the HF. The absence and/or failure of electricity at the HF during the night also discouraged many of the women who had come for delivery of their babies and they left. Mbarara used the LQAS results in family planning to improve their services in this area. Also, in monitoring immunization they realized that they needed to make more effort. They started educating the mothers and encouraging the fathers to take them to the HF. Kabale had their VHTs explain that health services were free and make house- to- house visits. They also integrated their immunization efforts with other health services to improve their score on this indicator. At the beginning of this report, we started with the assumption that there was a relationship between the degree of institutionalization and the effectiveness of health service delivery, at minimum, the percent of women who give birth in a health facility of some kind. This relationship can now be tested directly. If we add together the two scores on institutionalization in Figures One and Two and assign ranks, we can compute a Spearman s Rho with the rank ordering of the districts based on the data in Table 1. When this is done, the correlation is In other words, the combined institutionalization index 23

24 scores and the district rank are related, and that association is moderately strong. Given the above data on the number of examples of how LQAS was used, confidence about the causal connection represented by the correlation is established. Independent of quantitative measures, one can also supply some qualitative judgments as well if based on an accurate perception of the focus groups and interviews across multiple levels of analysis. On this basis, Hoima and Kaberamaido represent interesting contrasts. The data collectors and supervisors in the former instance were unenthusiastic about LQAS, with one person suggesting that the money could be better used to support health services while in the latter instance their counterparts in Kaberamaido were quite enthusiastic. There can be a number of reasons for these fundamentally different attitudes. First, the budgets for the data collection were sorely underestimated in Hoima where the conditions were quite challenging. For example, that area has particularly difficult access, the team had experienced a breakdown in transportation, multiple language skills were needed, and the area lacked overnight lodgings. These deficiencies may have led to resentment upon the part of those involved. Although the budgets for data collection are customized for each district, this specific LQAS funding came from SDS and probably the estimate was not informed by experience, especially the unusual conditions that existed. Secondly, Kaberamaido even without a second round of data scored a big success in increasing coverage with latrines in this district and this result empowered the supervisors and data collectors. Also, they used the LQAS results to obtain additional funding from CSF for HIV/AIDS work. This is one of the critical points about why a second round of data collection can be so crucial in the process of institutionalizing LQAS. Although Hoima has made a number of attempts to use the LQAS data on the basis of the reports that are available, they have not had any measurement of whether their changes in strategies and tactics have had a pay- off in contrast to Kabermaido. One implication of this tale of two districts is the importance of ensuring that there are always at least two rounds of LQAS that are supported so that changes on the indicators can demonstrate the efficacy of this methodology and provide the data collectors and supervisors with a sense of empowerment. Third, a subtle factor that was suggested by Charles Nkolo is that Hoima is an old district whereas Kabermaido is a relatively new one. There is a large literature on age of organizations and resistance to change that might be applicable in explaining differences between these district organizations. However, this issue would require additional research. Implied in the discussion of these two districts is one of the major reasons for the success of LQAS. Not only does it highlight with its system of red flags sub- counties and districts that are not performing well but also when changes in strategies and tactics are made, the resulting improvement in the effectiveness in the delivery of services reinforces the trust in 24

25 this monitoring and evaluation system and gives health management a sense of efficacy. But it is not only feedback about the improvement that accounts for the success of this system. There are in our opinions and based on the information received in the interviews three important ingredients that tend to reinforce each other. First, when the preliminary results are provided, all the key stakeholders are invited. This is defined not only by the key political and technical leaders in the districts such as the Local Council V Chairperson and the Chief Administrative Officer, the Sub- County/Parish chiefs, the implementing partners, religious leaders, the media among others. Given this diversity, the district feedback session mobilizes support for changing the strategies and tactics and providing better services. In this regard it is instructive that the terminology used in this setting (e.g., stakeholders) is associated with business schools in the United States. It is a concrete example of how to create ownership in a developing country (USAID, 2013). Second, the villages outside the urban areas appear to have high social capital. The definition is the extent of cooperation within the village. As reported in Appendix A, it became apparently in the focus groups, that there was not much variation. Most sub- districts or counties had a high degree of cooperation as illustrated in community members sharing transportation, helping in funerals, and aiding in planting. In the original research design visits to villages had been planned on the assumption that they varied on this dimension (see Appendix A). But since all the data collectors and supervisors reported a lack of variation, this part of the research design was eliminated. This high level of social capital makes it much easier to develop effective strategies for social services. Naturally the incorporation of the sub- county / parish chiefs in the dissemination of the results reinforces the extent of cooperation. Third, and described below, is the system of how the VHTs are selected and their procedures for encouraging women to have antenatal visits and to have their children immunized. Since they are elected, they have the trust of the community. They also rely upon the parish chiefs to help them in their campaigns to get women to visit the HF when they are available. And since the number of households for which each member of the VHT is responsible, only 25, they know when a woman becomes pregnant and can rapidly promote early antenatal visits. The measures of institutionalization discussed above are at the district level. Quite separate issues emerged when one moves to the national level. At this level, one has to observe that some 20% of the districts in the country have had no exposure to LQAS. Based on the evidence compiled in Figure One, one observes that some districts need more experience with LQAS before one might describe the monitoring and evaluation process as institutionalized in all districts where it has been introduced. If one accepts the cut- off of a 25

26 score of nine or 75 percent on the first index of institutionalization as suggested above, then the districts of Mbarara, Kaberamaido, Hoima, and Tororo require additional training inputs. However, what should be exceptionally encouraging to USAID is that of these four districts, all but two, Kaberamaido and Hoima, scored highly on the utilization of the data in planning and in these two latter cases, other data obtained from Lydia, the NLF in Hoima and from Esther Sempira, the national coordinator of STAR- E LQAS indicate that Hoima has been active in using the data in planning. This topic leads naturally into the discussion of challenges. 26

27 CHALLENGES The first challenge is to extend institutionalization not only to the four districts indicated above, but also another ten districts that, like Hoima and Kaberamaido, did not receive funding in From our list as annotated by Esther Sempira, STAR- E LQAS national coordinator, the other districts are: Bundibugyo, Masindi, Buliisa, Adjunami, Kotido, Koboko, Nkapiripirit, Kiboga, Mubende, Mukono, and Rakai. Beyond this are the 32 districts that have never had any exposure to LQAS. But as indicated in the recommendations the first priority should be these 13 that is the list of 11 plus Hoima and Kaberamaido to solidify the previous investment in LQAS before one extends this monitoring and evaluation process to another 32 districts. One of the unexpected challenges that emerged in the data collection from the DHOs and the Assistant DHOs was the problem of turnover in these positions. Recent appointees who arrived after the last round of data collection did not know about LQAS and seemed uninterested, as can be seen in Table A.2. This poses a long- term threat to the institutionalization of the system if members of top management teams are uninterested in receiving the reports and they are unlikely to be interested if they had not received any exposure. Thus to sustain the LQAS system one important issue is to continue to expose management to this system of monitoring and evaluation. To ensure that future DHOs and Assistant DHOs receive some training in the advantages of LQAS and possibly LQAS principles, necessitates some arrangement between the Ministry of Local Government and the Ministry of Health so that the selected individuals for promotion or those newly recruited into these positions can receive this training The data collectors and supervisors generally agreed that the major challenges to data collection were the problems of collecting data from quite busy people and the need to return to the households selected from a table of random numbers. These issues are connected with the need to sample specific age categories for disparate sections of the LQAS instrument. Another common complaint was the need for more time to collect the data. Some data collectors reported that they felt stressed by the time pressures and having to work sometimes on both Saturday and Sunday to finish their data collection, sometimes possibly without additional facilitation. Compounding the time pressures were the distances involved in reaching different households and a variation on this theme from supervisors was the difficulty of supervising two data collectors who were working in quite disparate parts of a village or community. Magnifying the time pressures and distances were reports that the transportation provided broke down or was not readily available. Another obstacle mentioned by supervisors was a lack of time for data analysis. One complaint in a district with only one round of LQAS data collection was a problem of obtaining data during the rainy season. Other districts noted that this problem had been solved in their second round of data collection. As indicated in the recommendations for sustainability, these problems 27

28 can be solved easily with planning that responds to the needs of the districts. Learning was also demonstrated in a district that reported at first some resistance from Muslims about certain questions regarding sexual behavior. But with the support of the local leaders, these issues were resolved. One of the anticipated challenges, namely, problems of collecting data across 59 indicators posed no problem. Two challenges for the future of LQAS are the age of either the data collectors or the supervisors. Four individuals were 50 years of age or older. These individuals will eventually have to be replaced by, perhaps, younger individuals. This Human Resource matter ought to be discussed with district managers. Another problem is that in some districts, not much diversity in the positions occupied can create difficulties in acceptance (see Table A.3). While it is understandable that the health sector would supply many of the individuals who were trained, there were few from the education sector, one sector that the Director, Local Government Administration wants to emphasize. Ideally one would like to have data collectors and supervisors that reflect all the relevant sectors in a district to build more support for the methodology. In Tororo, six of the data collectors and supervisors were from the community development office, one was from planning, which is akin to the former, and two were health assistants. This concentration may diminish acceptance in the future. Two more subtle issues emerged had not expected. The first is the expectations of individuals interviewed that they would receive something for participating. Apparently this became particularly heartrending when interviewing orphans and other vulnerable children. Closely connected to this first issue is a second one, namely, the lack of dissemination of the results back to the sub- counties, parishes, villages through the political and technical leaders and also the VHTs. Three districts mentioned this problem, Bushenyi, Kabale, and Kamwenge, but at the same time indicated that they did not have the funds to ensure dissemination. This would appear to be an easy problem to resolve as long as one avoids expensive printing procedures. Another challenge, particularly in the eastern region was the problem of the lack of availability of HF as well as an unstable supply of electricity in the HF that affected the willingness of the pregnant women to give births there. In particular, there is a dearth of HF in the national park located in the eastern region. Also, the unevenness of the supply of electricity was affecting the quality of the vaccines being maintained in the refrigerators. Beyond this, many of the heath facilities are not connected to the electricity grid and either rely upon solar or paraffin/wax candles. These problems help to explain to a certain extent the negative correlation between visits to HF and recognizing danger signs in children. But as suggested above, the challenge may be the inability to pursue multiple improvements in social services given limited budgets. How much this was a problem in the districts that was not explored. Nor was the particular emphasis in a district was influenced by the concerns of the major implementing partner investigated either. 28

29 The research did examine the impact of the availability of HF in a specific district and how this relates to the indicator reported in Table 1, namely, the percent of women having their delivery in a health facility. The availability of HFs is a function both of the number of square kilometers and the number of people who live in a specific catchment area. The number of facilities for each district per 1,000 people living in 100 square kilometres is reported in Figure Three. The construction of this table requires two assumptions, neither of which is true. First assumption: HFs are evenly distributed across the district. As indicated above, this is not the case with particular areas such as national parks lacking these facilities. Second assumption: The proportion of women in child- bearing ages are approximately the same relative to the population across all nine districts. Again, this is certainly not true and especially in the districts that have more urban areas. STAR- E LQAS analysis revealed that mothers having their births in a health facility, is correlated (r = 0.61) with the number of antenatal visits, as one would expect. But if one accepts these limitations, Figure 3 still does reveal interesting findings. As can be observed there are quite strong variations among the districts with Hoima in particular having a very low number, one- third of a facility per 1,000 persons in a 100 square kilometre catchment area. At the other extreme is Kabale, which has more than four facilities for the same size area and population density. If one ranks the availability of HFs and compares this to the rank order reported in Table 1, the Spearman Rho is a very high.79. In other words, there is a strong relationship between percent of women who deliver their babies in a facility and the availability of an HF. What is worth observing in Figure 3 is that Bushenyi, which has the highest percentage of women giving birth in HF, has the second lowest availability. This district represents a model for other districts and illustrates how the use of LQAS can allow for the effectiveness of services despite a considerable resource constraint. It also suggests one of the best ways of testing for the effectiveness of LQAS, namely whether it can overcome a major resource constraint. But at the same time, if one analyses all the districts in the STAR- E LQAS super- dataset, there is a surprising negative correlation between the use of HF for birth delivery and taking sick children to one. In this larger analysis, the eastern region has a significant negative correlation. In an interview with the Assistant DHO in one of the districts, she reported that the lack of electricity meant that they could not keep vaccines safe. 29

30 Figure Three The Number of Health Centers Relative to Population and Square Kilometres Not all challenges are necessarily negative ones. Several positive challenges emerged, positive in the sense of how various respondents felt about LQAS. One DHO wished that LQAS could be extended to additional service sectors because he wanted the support of other sectors of local government. Interestingly enough this was also echoed in the interview with Patrick Mutabwire, Director of Local Government Administration, who desires to extend the methodology particularly to the education sector. An Assistant DHO would like to have more individuals in the local government trained in the methodology. This idea might be considered when the transfer of knowledge across districts occurs (see recommendation below). On the relationship between the district government and the central government, several questions were included in the interviews with the DHO and the deputy DHO on the assumption that there might be problems in the use of LQAS data by the central government when it is making its five- year plans. One of these questions, namely the amount of technical support, was misconstrued and therefore was eliminated. Instead of answering about the support from the central government, the respondents reported on the support received by STAR- E. The two questions about whether the data had ever been transmitted and/or used by one or another ministry in planning could not be answered. The DHOs and the Assistant DHOs were unaware of how the data were transmitted and how it influenced the planning processes of the ministries of the central government. This then becomes a major challenge for ensuring that the LQAS process becomes institutionalized at the national level. Strikingly, one of the DHOs proudly announced: the districts owned the data. 30

31 Indeed, the strategy of MSH and LSTM was to encourage this attitude. But the downside is that this emphasis has not facilitated the movement of the data into the appropriate planning departments at the central government level. It might be noted that various ministries have invested in large surveys, in particular the Ministry of Health as it expands HMIS. The time frames and concerns are not the same and this poses some real challenges of how to overcome this blockage. Learning and Challenges As has already been demonstrated, LQAS produced a considerable amount of learning in the nine districts involved in this study. The three regional meetings organized by MSH reports are available from Alice Nakagwa of MSH (2013 Regional Information Sharing Meeting: Central and East Central Region Districts Kampala: MSH, Uganda is one example). Even more extensive is the summary report prepared by Esther Sempira, the national coordinator of STAR- E LQAS, that includes information on about 40 districts (see Appendix C for one year). These provide additional documentation of the successes of LQAS to impact on social service effectiveness. The districts recognized that they were not performing well in certain areas and then attempted to improve their scores. But data collected by LQAS are not the only form of learning that occurred in the nine districts. Two major external sources of learning were reported in the interviews with the DHOs and Assistant DHOs (or their substitutes in several cases, see Table A.2). and in more than one district. The most important was a change in the policy of the Ministry of Health to encourage the development and training of VHTs. A number of districts reported on the important change of introducing VHTs who are selected by the members of the village. In the discussion above about when the preliminary reports were made, frequently the decision was made to use these men and women to improve social services and in particular in maternal and child health. Furthermore, the procedures that the Ministry of Health used to establish this key component of the health care system needs to be carefully understood to appreciate why this is such an effective policy. Members of communities are involved in the selection of VHTs and vote for them. Therefore, the selected men and women are trusted and their recommendations are more likely to be taken. One member as has already been stated is responsible for 25 households and the number of team members is usually between four to six since most villages are quite small. Indeed, it is a contention that part of the success of LQAS relies upon the way in which these women and sometimes men could be utilized given the recognition of some failing such as the low rate of antenatal care or the rates of immunization or tests for HIV/AIDS. As can be seen above, the VHTs sensitized mothers to various issues, educated them in the importance of integrated disease management (IDM), the recognition of danger signs in small children, the importance of antenatal visits, etc. 31

32 A second example of learning was the intervention of Baylor College of Medicine in Kamwenge and Kabarole to provide more rigorous training of the VHTs. In addition, they paid VHTs 25,000 shillings (approximately $10), and gave them umbrellas and bicycles as well as provided districts with ambulances. They also established a voucher system of 5,000 shillings that allowed mothers to take motorcycles to visit HF for their antenatal visits. Baylor also provided pregnant mothers with bed sheets, soap, gloves, and baby shorts. The consequence was a very large increase in antenatal visits and giving births in HF. But the intervention was for only one year and since then percentages have started to drop as the voucher system for visits no longer exists. Thus, this system cannot be maintained but perhaps some elements might be sustainable. This is a significant challenge. Relative to the question posed at the beginning of this report of whether learning that has improved the effectiveness of services is a combination of both LQAS and of various interventions by either other donor agencies and/or ministries, the answer is that indeed it is. It is impossible on the basis of the data available to estimate what proportion of changes in the effectiveness of social services is attributable to one or another learning process. The single exception was the striking increase in the use of HF given the Baylor intervention, but as already noted, this increase was unstable. Relative to other forms of learning, some of the DHOs did report participating in various meetings that increased their learning. A national conference on family planning organized by the Ministry of Health was mentioned. In some regions there are regular meetings about maternal health. And of course, there have been meetings about HIV/AIDS. MSH also organized three regional meetings but only one of our respondents mentioned it, an NLF. Sustainability and Challenges The first strategy explored in the interviews with the DHOs, Assistant DHOs and the focus groups was whether the data collection could be integrated into the on- going work of the supervisors and the data collectors. There was almost total consensus that this could be done. Quite surprisingly few mentioned the loss of the per diem that they received when collecting data. The area that requires some expenditure of funds- - that was repeatedly mentioned as indicated above in the discussion of challenge- - is the adequacy of transportation. Therefore, if LQAS is to be sustained, this is a major issue, not only because of the necessity of visiting villages that are remote but also because the equipment provided is inadequate. From the perspective of the DHOs and their Deputies, a typical response was: Why Not? And some indicated that there were active discussions about how this might be accomplished. 32

33 Although there was consensus that data collection could be integrated, one word of caution is necessary. As can be observed in Table A.3, there is considerable variety in the positions of the data collectors and their supervisors. One might imagine that this would be more easily achieved with some positions, e.g. those who work in the health sector, than with other positions. Another word of caution came from a newly appointed DHO who was familiar with the process because of a previous appointment. He thought that there might have to be some diminution in the quality of the data so as to reduce costs. Among other options he considered were a reduction in the number of indicators, less supervision of the data collection and fewer tests of its quality because the double interviewing of the same respondents would no longer be conducted. However, other members of the health management team stressed the importance of the many indicators as a way of building support within their district governments for the process. Several of the challenges outlined above represent opportunities not only for sustainability but also for improving the quality of the collected data. Lengthening the amount of time in which the data is collected leads to more successful integration of this process with the on- going work of the supervisors and data collectors. More critically, it would reduce the stress encountered because of the problems of certain age groups not being available at particular times and the need to collect all the data in one week. However, much the integration of data collection into the work of the data collectors and supervisors would appear to be a solution to sustainability, it requires carefully planning; see the recommendation below. The second strategy, namely, the transfer of knowledge, to those districts with only one or two rounds of data collection and even more to those districts with no rounds, also received considerable support. All the data collectors and supervisors believed that they were well trained enough that they could train others. The previous two strategies were strongly endorsed but the third one was roundly rejected. In this strategy we posed the possibility that LQAS could be phased out when the percentages reached a certain level with the idea that the health care system would systemically collect this data. In particular, we used the examples of births in HFs. We had not considered asking questions of the health management team about their commitment to LQAS because of the conviction we would only obtain political correct answers since the whole interview was about this methodology but in fact this question accomplished this very objective. All the DHOs and Assistant DHOs, who had been exposed to LQAS felt that even if the percentage reached 100%, it would still be necessary to have this information because the mothers and the children are constantly changing and there was no guarantee that the percentage would not decline. Several mentioned how important were the red flags to know which particular parishes or sub- districts needed special attention. 33

34 As indicated in the discussion of the research strategy (see Appendix A), the primary objective of this research was to measure institutionalization and sustainability at the level of the district and not the central government. Certainly as indicated in the discussion of the challenges, the connection between the districts and the central government remains a challenge that needs to be considered. However, we did conduct a few interviews at the central government level, in particular with Patrick Mutabwire, Director of Local Government Administration. He chairs a technical working group (TWG) to focus on the sustainability of LQAS and even its expansion to include more indicators in sectors of concern. He stressed the importance of schooling and the failure of the public sector to provide adequate education. Also, in an interview with Esther Sempira, the national coordinator of STAR- E LQAS, she reported that the Ministry of Gender, Labor and Social Development has found the data to be quite useful. Although a number of efforts have been made by MSH to sensitize the various ministries to the importance of sustaining the collection of data by the LQAS method, the question remains as to whether a new strategy can be devised. One possibility is to identify those individuals in the central government that have been responsible for organizing workshops on various subjects, not just in the Ministry of Health but in other ministries as well. Implied in this idea is that those individuals who have done this are the ones most interested in solving problems and therefore most open to examining the indicators that are being measured as the source of new kinds of workshops that represent attempts to solve the problems reflected by the low scores on specific indicators. These individuals might be recruited to serve on the TWG being formed in the Ministry of Local Government. Since in these ministries communications with these individuals must go through the head of the ministry, the justification for their serving in the TWG is the possibility that if several if not all ministries work together they would be more effective in raising funds for LQAS from various donor agencies in the world. 34

35 RECOMMENDATIONS From the above findings several recommendations about increasing institutionalization and sustainability while reducing challenges emerge. At the request of STAR E- LQAS, these recommendations are grouped into two categories. First are the short- term recommendations that can be implemented in the next nine months while funding remains depending upon the amount of funding still available. If funding is not available for recommendation two, then this would move into the category of recommendations for the long- term, which are recommendations requiring additional sources of funding. But although they might require additional funding the recommendations being made consider a number of ways in which the costs can be reduced. Short- Term Recommendations 1. Attempt to include members of other ministries in the Technical Working Group being formed by Patrick Mutabwire, Director of Local Government Administration. Considerable efforts have been made by STAR E- LQAS to find individuals in other ministries that are strategically placed and interested in sustaining LQAS. Certainly the Ministry of Gender has found the data collected to be useful for their planning. One potential source of champions for LQAS in each of the ministries relevant to the sources of data being collected across the 59 indicators are those individuals who have organized workshops for the districts. The data in these indicators suggest topics about how social services can become more effective via specific kinds of interventions that they might be willing to support. Once they express an interest in this, then the next step would be to request their participation in the TWG. As the number of ministries that participate in the TWG increases, then sharing the costs of LQAS becomes possible, which in turn reduces the amount that any one ministry would contribute. 2. Support a second round of LQAS data collection in the 13 districts that so far have had only one round. The long discussion of the differences in the reactions of Hoima and Kameramaido above indicates the importance of having feedback when strategies have been changed so that the district becomes more committed to the data collection process. Since one round has been accomplished and the data collectors and the supervisors in at least the two districts that I visited think that they have the necessary skills, it would be unfortunate to lose the momentum that has been achieved. This would also reinforce their level of skill as well as provide additional information for planning in these districts. 35

36 Long- term Recommendations 1. Hold planning committees among districts with similar sets of challenges to discuss how best to integrate the data collection process in the on- going work of the supervisors and data collectors in those districts. As indicted in the research findings, the careful integration of LQAS data collection with the work of the data collectors and respondents allows one to reduce stress and improve quality as well as perhaps address some other issues. For example, the budgets used to support normal health service delivery could support most of the transportation costs, which precludes having a separate LQAS budget. These meetings should involve the different sectors in the district that would find the data useful, i.e. more than just the health sector, so that there is commitment built in the district for this exercise. 2. Begin to train new data collectors and supervisors in those districts that as yet have not had any exposure to LQAS. A major resource is not only those supervisors and data collectors that have participated four times but also the NLFs that have been trained. This elite group should be involved in the planning and transfer of knowledge across districts. But when decisions are made to transfer knowledge to new districts, it would be important to include in the planning the assignment of enough funds to ensure two rounds of data collection even if they should be two years apart. As indicated in the previous recommendation, this is necessary to build support within the district for this process. 3. Every several years train the newly appointed DHOs and Assistant DHOs in the LQAS process; if resources permit, retrain all district managers to reinforce LQAS as a part of the planning process. This might be easily combined with the second recommendation depending upon how quickly the roll- out of the transfer of knowledge takes place. In any case, it is important that this exposure be continued in the districts that have already utilized LQAS. In summary for little cost, LQAS can be sustained and extended to those districts that have not yet had either enough experience with the system or none. 36

37 APPENDICES Appendix A: Discussion of the Research Design The selection criteria for both regions and districts within them were as follows. The central region was excluded because of the high rate of urbanization and development. In contrast, the northern region was excluded because a major USAID project, NU- HITES, started only in 2012 and will continue until Therefore, evaluating institutionalization and sustainability is premature. This left the western region, the south- western region, and the eastern region. At the time several other regions such as the Nile or northeast regions were not suggested. The choice of districts was designed to maximize variation within regions. Table 1 indicates that despite the interpretation that there are major differences between regions, there is also considerable variation within them as well. Although this table uses data collected in 2012, four of the selected districts also collected data in The five exceptions are Kabarole, Mbarara and Tororo, whose implementing partner, Sunrise- - OVC, did not have enough money to fund another round of data collection, and Hoima, which had funding for only one year from the CSF. The other exception is Kaberamaido, where the next round of LQAS is scheduled for Indeed, twenty districts out of the 72 reported in Table A.1 did not have funding in Another consideration in the research design was the possibility that there were also major variations across the sub- districts. The LQAS system of sampling households in villages is designed to capture this distinctive form of variation and signal it with red flags to indicate where specific problems exist. Both the data collectors and the supervisors were asked to report if significant differences existed within parishes or sub- counties in the areas in which they worked. The reasoning behind this was that perhaps there was more civil society in some local communities than others, which could be important for overcoming challenges and increasing the effectiveness of service delivery. A few significant differences emerged in the context of questions about the obstacles to collecting LQAS data. But the more important issue is whether across these small areas variations in the amount of civil society or willingness to cooperate exists. This is of special interest given the SDS fund as well as the Community Health Alliance Uganda (CHAU). In most cases, both the data collectors and supervisors agreed that there was not much difference in the extent of cooperation. Neighbors are cooperative and help each other with funerals and transportation. However, the few exceptions were of some interest. One community in the eastern region where Plan International had been working for some time was considered very different from the other communities. This suggests that in future research the number of red flags or their absence, the gold stars, in small areas be connected to specific interventions of organizations such as Plan, International, World Vision, UNICEF or a very important one that emerged in the data collection, Baylor Medical School training of VHTs. A common theme in the four cases that 37

38 should be explored in future research on the efficacy of the LQAS methodology is whether their interventions build an even stronger civil society because they provide programs for children. As a consequence, it is possible that people in the villages respond more to the various initiatives of the district office to improve their health, especially the support of VHTs. In other words, are the VHTs more effective when those INGOs or international projects emphasize children s well being? One district, Mbarara mentioned the differences between urban and rural areas in the extent of cooperation, which is to be expected. Since there were no major differences between sub- districts, this data is not reported. However, it should be understood that the degree of cooperation becomes a hidden resource that allows the collection of LQAS data as well as health district interventions such as sensitization by the VHTs to be more effective than they would otherwise be. At various points in the interviews the importance of being able to obtain the cooperation of the village chiefs was cited. Although sub- district variation does not explain differential effectiveness of LQAS, it is important for international donor agencies such as USAID to understand that the success of LQAS in Uganda may not be easily transferred to other countries where cooperative behavior is low. But while the concentration of data collection was on the district level of services, the central government was not ignored. Rather than ask questions in the central government looking down on the districts, the approach was to ask questions in the districts about how they interfaced with the central government in the use of LQAS data. This would be in alignment with the decentralized model of the Ugandan government. Several interviews were also conducted at the central government level to obtain some additional information about the challenges of sustainability. Table A.1 Percent of Women in Districts Giving Birth in a Health Facility By Year of Data Collection and Region District Percent Sample size 2012 Western Buhweju Buliisa Bundibugyo Bushenyi Hoima Ibanda Isingiro Kabale Kabarole Kamwenge

39 District Percent Sample size Kanungu Kasese Kibaale Kiruhura Kisoro Kyenjojo Masindi Mbarara Mitoma Ntungamo Rubirizi Rukungiri Sheema Northern Adjumani Apac Arua Koboko Kotido Nakapiripirit Nebbi Eastern Amuria Budaka Bududa Bukedea Bukwa Bulambuli Busia Butaleja Jinja Kaberamaido Kapchorwa Katakwi Kibuku Kumi Kween Manafwa Mbale Pallisa Sironko Soroti Tororo Central Kalangala Kayunga Kiboga Luwero Lwengo Masaka Mityana Mpigi Mubende

40 District Percent Sample size Mukono Nakasongola Rakai Ssembabule Western Buhweju Bushenyi Ibanda Isingiro Kabale Kabarole Kamwenge Kanungu Kasese Kiruhura Kisoro Kyenjojo Mbarara Mitoma Ntungamo Rubirizi Rukungiri Sheema Northern Arua Nebbi Eastern Budaka Bududa Bukwa Bulambuli Busia Butaleja Jinja Kapchorwa Kibuku Kumi Kween Manafwa Mbale Pallisa Sironko Tororo Central Kalangala Kayunga Luwero Masaka Mityana Mpigi Nakasongola Ssembabule

41 District Percent Sample size 2010 Western Buhweju Bushenyi Ibanda Isingiro Kabale Kanungu Kasese Kiruhura Kisoro Mitoma Ntungamo Rubirizi Rukungiri Sheema Eastern Budaka Bududa Bukwa Busia Butaleja Kapchorwa Mbale Pallisa Sironko Eastern Busia Butaleja Pallisa Sironko Western Bushenyi Kabale Masindi Northern Arua Lira Eastern Kamuli Mayuge Mbale Central Kampala Kayunga Masaka Mukono Western Hoima Kabarole

42 District Percent Sample size Northern Apac Kitgum Moyo Nakapiripirit Eastern Jinja Kaberamaido Soroti Tororo Central Kalangala Western Bushenyi Kabale Kamwenge Kyenjojo Masindi Mbarara Northern Arua Lira Eastern Iganga Kamuli Mayuge Mbale Sironko Central Kampala Kayunga Masaka Mukono Rakai Wakiso

43 Figure A.1 The Location of the Selected Nine Districts 43