THE ESCHOOL 360: AN EVALUATION OF IMPACT NETWORK S EDUCATION PROGRAM IN RURAL ZAMBIA

THE ESCHOOL 360: AN EVALUATION OF IMPACT NETWORK S EDUCATION PROGRAM IN RURAL ZAMBIA

EXECUTIVE SUMMARY THE ESCHOOL 360: AN EVALUATION OF IMPACT NETWORK S EDUCATION PROGRAM IN RURAL ZAMBIA In September of 2000, the Millennium Development Goals were created, partly to ensure that by 2015, children everywhere would be able to complete primary schooling. Much progress has been made, but there are still nearly 58 million children around the globe that are out of school. Those who do enroll in primary school often do not succeed. Compounding this, the governments in many developing countries do not have the resources to provide a quality education to all children. To address these issues, Impact Network designed the eschool 360, a holistic program operating in rural Zambia that seeks to provide a quality education while keeping costs at a minimum. The program provides locally-hired teachers with a tablet and projector to deliver elearning lessons. Teachers receive weekly professional development to enhance their pedagogical skills and ensure effective use of the technology. In 2013 and 2014, American University conducted an evaluation of the eschool 360. Researchers examined first and second graders at five Impact Network schools, five nearby government schools, and five nearby community schools. The evaluation consisted of a baseline data collection, qualitative research based on interviews and focus groups, cost research, and an impact evaluation measuring students math and literacy skills after 18 months. Researchers concluded that Impact Network schools cost 70% less than government schools, while improving the numeracy and literacy skills of students. Their findings suggest that: Impact Network serves students who are more disadvantaged than government schools. Impact Network serves students who enter school less academically prepared and younger than students entering nearby government schools. Compared to government school children s households, Impact Network s students come from significantly larger, poorer, and less well-educated households. The eschool 360 program was implemented successfully at Impact Network schools. While some schools experience power outages, and a small number of teachers struggle to use the technology, overall the program is operating as intended. Parents perceive the elearning program at Impact Network schools as an improvement to overall schooling quality. Impact Network provides important services that improved the overall working conditions for teachers, so that they are more motivated to attend school and teach. Teacher attendance is higher at Impact Network schools than government schools and community schools. Across all subscales on standardized EGMA and EGRA tests, Impact Network students demonstrated learning over the 18-month period. Impact Network students show improvement on a standardized EGMA tests measuring mathematics, and EGRA tests measuring literacy. In most subscales, Impact Network students learn as much or more than their counterparts at government and community schools. Impact Network s schools cost about 27 percent of government and 29 percent of community schools on an annual per student basis. Not only does the eschool 360 program improve students academic achievement, it does so in a more cost efficient way than the established schooling institutions. Impact Network is using the results of this evaluation to document the eschool 360 model and share insights with policymakers in Zambia. Additionally, they have recently established a partnership with American Institutes for Research (AIR) to develop new interventions and strengthen existing programs before expanding the eschool 360 to additional schools.

In September of 2000, world leaders came together to commit their nations to help reduce extreme poverty, and adopted what is known as the Millennium Development Goals. The second of these goals was to ensure that, by 2015, children everywhere would be able to complete their primary schooling. Across the globe, thousands of schools were built by charities and NGOs to help meet the goal of achieving Universal Primary Education. Despite this, there are still nearly 58 million children around the globe that are out of school. 1 Even those who do enroll in primary school do not often succeed. In Sub-Saharan Africa only 53% of students who enroll in primary school complete it, and only 28% go on to enroll in secondary school. Less than 10% make it to university. Poor children living in rural areas are even less likely to succeed than their wealthier urban counterparts. 2 Many governments do not have the resources to both build schools and ensure that they provide a quality education to all children. This gives rise to community schools that are often built by NGOs and then handed over to local communities to operate. But without funding, trained teachers, management, supplies, and a rigorous curriculum, these community schools often fall into disrepair. One program in rural Zambia, called the eschool 360, seeks to resolve both of these constraints to provide a quality education while keeping costs at a minimum. The program provides locally-hired teachers with a tablet and projectors to deliver elearning lessons. Teachers are provided weekly professional development focused on helping them use the technology and enhance their pedagogical skills. There are 10 such eschools in Zambia currently, run by a New York City-based non-profit called Impact Network. Impact Network partners with ischool, a Zambian organization that creates elearning content designed to move the country s educational system away from rote memorization towards activity-based programming. American University conducted an evaluation of the eschool 360 from January 2013 to July 2014, including quantitative, qualitative, and cost-effectiveness analyses. The team consisted of Paul Winters, Associate Professor in the Department of Economics and Maja Schling, a PhD candidate in the Department of Economics. The evaluation examined five Impact Network schools, as well as five surrounding government schools, and five other community schools. Researchers concluded that Impact Network schools cost 70% less than government schools, while simultaneously improving the numeracy and literacy skills of students. This report provides an overview of the eschool 360 program, describes the evaluation design, and summarizes the findings. It concludes with recommendations. THE ESCHOOL 360 PROGRAM The eschool 360 is a holistic technology solution for schools in underserved, rural 1

areas. The program provides a tablet, loaded with activity-based lessons, and a projector to classrooms. It seeks to empower local teachers with creative lessons, lesson plans, and regular, ongoing training. The result is a comprehensive solution to deliver high quality education year in and year out, in a cost-efficient manner. The program contains many integrated components, described below: elearning hardware: Teachers use a low-power tablet and projector to assist instruction and manage group work. Standardized curriculum: ischool developed lessons and lesson plans to move away from traditional rote learning, while remaining aligned with the national standards of Zambia. Teacher training: Teachers are hired locally. While untrained, this provides much needed job creation in the rural setting and ensures that teachers are vested in the communities. To improve their skills, each teacher receives weekly observations and feedback from experienced teacher supervisors, and ongoing professional development. Weekly management: Professional staff support every school, providing accountability and logistical support. Community ownership: Ownership is fostered through community activities. Additionally, each school s PTA supports the school and actively engages parents in the school. School supplies: The government does not provide supplies to community schools. Since the program targets under-resourced families, students and teachers are provided with adequate school supplies. Teacher salary: In other community schools, teachers are often volunteers who are paid sporadically and poorly. In the eschool 360 program, teachers receive a living wage. Solar: A reliable, environmentally-friendly solar electric system is required in most rural areas to power elearning and lighting. Building care: Respectable and safe facilities are key to providing a quality learning environment for the students. Rural security: A safe, steel doors and a security guard are provided. The program is able to provide all of this for an estimated $3 a month per student including all of the above components. Impact Network seeks to have every donor dollar make a difference, and establish a model whereby governments, communities, and NGOs can increase the quality of their schools without breaking the bank. 2

THE EVALUATION The preferable approach for an impact evaluation is a randomized trial, where eligible students are randomly assigned to a program and control group. This creates two similar groups, whose achievement can be compared as a way to see how much students in the program learned, compared to how much they would have learned without the program. However, such a design was not feasible for this program evaluation, due to limitations around scale and local conditions. Thus, a non-experimental evaluation was completed. The team from American University worked with local evaluators to collect baseline and follow-up data from three different groups of students: 1. Impact Network schools 2. Government schools in the surrounding area 3. Other community schools in the surrounding area Each group consists of five different schools, with 40 students in the first and second grade respectively, for a total of 15 schools and 600 students. The government schools and other community schools comprise the comparison group for the analyses. The evaluation consists of the following main components: A comprehensive baseline description of students across all 15 schools. Students were measured on basic math and literacy (EGMA and EGRA tests), and demographics before the school year began. 3 A qualitative study based on interviews and focus groups with parents, teachers, and staff across all 15 schools. Based on these interviews, insights on changes in the quality of education and school services were obtained. A cost study to ascertain the true costs of educating children in Eastern Province every year. The yearly cost per student was determined after surveying each school and obtaining cost information from Impact Network and the local branch of the Department of Education. An impact evaluation. Students basic math and literacy skills were measured 18 months after the baseline measures. Researchers compared these measures to the baseline measures, to determine the learning gains of Impact Network students. These progress measures were then compared to students in government and other community schools in the region. THE KEY QUESTIONS The impact evaluation examines several questions to determine whether the eschool 360 program is meeting its goal of providing a quality education at a low-cost. The questions include: 3

1. Does the provision of a solar panel system, lesson plans, the introduction of the ischool curriculum, as well as professional teacher development improve overall student attainment in numeracy and literacy skills? 2. Is the eschool 360 program a cost effective way of providing quality education? 3. Does the eschool 360 program lead to an improved perception of school and education quality among students, teachers, and parents? The remainder of this report summarizes findings from across the baseline study, qualitative study, cost study, and impact evaluation to answer these five questions. BASELINE FINDINGS families with little previous access to education has been successful. Households with children in Impact Network schools are not very different from households with students in other community schools. Impact Network students face similar living conditions and challenges as other community school students. Students attending Impact Network schools are significantly younger than students in government schools and students in other community schools. This implies that ontime enrollment is positively affected by the program, and could ultimately lead to positive effects on student performance. Data on student and household characteristics were collected for students in the first and second grade. Household data collected included measures of wealth Impact Network is and locality, and student data serving students included performance measures who come into their on literacy and numeracy skills. schools less prepared than the Main findings from the baseline students entering evaluation found that 4 : nearby government schools. Households with children in Impact Network schools are significantly larger, poorer and less educated than those in government schools. This provides evidence that Impact Network s strategy of targeting poor Impact Network and community schools serve students who enter school with lower numeracy skills than government schools. For the most part, there were no differences on the EGRA subscales at baseline, because literacy levels were universally low. Thus, Impact Network is serving students who come into their schools less prepared than the students entering nearby government schools. QUALITATIVE FINDINGS The qualitative study was conducted in June and July of 2013. It included key informant 4

interviews with head teachers and teacher supervisors employed under the program as well as focus groups with first and second grade teachers, and randomly selected parents. Interviews and focus groups were conducted at all 15 schools in the study. Based on these interviews, researchers obtained insights on changes in the quality of education and school services, as well as perceived changes in student and teacher behavior. In general, the qualitative evaluation found that 5 : Parents across all schools appear to value education highly and believe that sending their children to school will not only enable them to have a better future, but will also benefit the family and community as a whole. However, families are constrained by several factors, including severe and widespread poverty among households, difficult climatic conditions, and the persistence of traditional practices that inhibit school attendance among children. Findings confirm that program components were implemented successfully at Impact Network schools. While some schools do experience power outages and computer issues, and a small number of teachers struggle to use the technology, overall the program is operating as intended. Parents perceive the elearning program at Impact Network schools as an improvement to overall schooling quality. They observe greater motivation and better performance in their children. Impact Network provides important The elearning program services that have seems to lead to an improved the increase of motivation overall working among students in Impact Network schools. conditions for teachers. Households seem equally involved in school activities across school types, indicating that the elearning intervention induces PTA participation but not at a rate greater than other schools. Impact Network provides important services that have improved the overall working conditions for teachers, so that they are more motivated to attend school and teach. While they are still not being paid as much as government employees, untrained teachers employed at Impact Network schools receive relatively high salaries and are paid regularly. Infrastructure is improved and more materials are available. Teachers also receive ample support through trainings and weekly observation. Smaller class sizes further improve teaching conditions. Teacher attendance is higher at Impact Network schools than government schools and other community schools. Impact Network has a 5

strict attendance policy in place. Overall, this qualitative survey paints a positive picture of the eschool 360. The findings indicate the eschool 360 program can create real change in the way students are taught. COST ANALYSIS FINDINGS The cost analysis attempts to ascertain the true costs of educating children in Eastern Province each year. School costs are divided into four different types of ingredients: personnel, facilities, equipment and materials, and client inputs (that is, the costs of students and their families). 6 Other costs of the Impact Network program that do not apply to government and community schools are categorized as program-specific costs. After surveying each school and obtaining cost information from Impact Network and the local branch of the Department of Education, the yearly cost per student is calculated. The main findings of the cost study are detailed in Figure 1 and below 7 : Impact Network has succeeded in creating schools that cost about 27 percent of government and 29 percent of community schools on an annual per student basis. Impact Network s schools spend less on each cost ingredient, except for equipment and material costs. Parent or household costs were considerably lower at Impact Network schools compared to government and community schools. On average, parents at Impact Schools pay less than $1 per child per year or about 2% of what parents at government and community schools pay. The high cost of educating a child in government and community schools Cost per S tudent (in US D with exchange rate 5 ZMK : 1 US D) 140 120 100 80 60 40 20 - F igure 1: Cost per S tudent by S chool Type Impact Network S chools Government Schools Community S chools Personnel Facilities Equipment and Materials Client Program-Specific 6

is due to the considerably higher salaries that are paid to government-trained teachers there. Overall, this indicates that if the quality of education at Impact Network schools leads to similar educational outcomes as government schools, it will do so at a far lower cost and thus be much more efficient. 18-MONTH IMPACT FINDINGS The 18-month follow-up was conducted in July of 2014, and found 8 : Across all subscales on standardized EGMA and EGRA tests, students demonstrated learning over the 18-month period. The increase in learning ranged from 3 percentage points to 37 percentage points for Impact Network schools, 0 percentage points to 41 percentage points for government schools, and 1 percentage point to 35 percentage points for community schools. After 18 months, Impact Network students show improvement on the standardized EGMA test measuring mathematics. In most subscales, Impact Network students learn more than their counterparts at government and community schools. Figure 2 shows the percentage point increase in test scores for the EGMA subscales from baseline to follow-up, including an overall 25 percentage Figure 2: EGMA Score Increases Between Baseline and 18 Month Follow-up 45 40 41 35 37 35 35 37 37 Percentage Point Increase 30 25 20 15 25 21 19 15 33 27 18 32 23 28 21 28 19 17 30 10 11 9 10 5 5 5 5 - EGMA Total Oral Counting Rational Counting - Number Recognition Quantity Discrimination Pattern Completion Word Problems Addition Problems Subtraction Problems Impact Network Government Community 7

45 Figure 3: EGRA Score Increases Between Baseline and 18 Month Follow-up 40 35 Percentage Point Increase 30 25 20 15 16 10 12 11 12 5 0 8 7 7 7 6 5 4 4 3 1 1 EGRA Total Letter Sound Knowledge Non-Word Decoding Oral Passage Reading Listening Comprehension Impact Network Government Community point increase in numeracy skills. After 18 months, Impact Network students show some improvement on a standardized EGRA tests measuring literacy. However, the overall literacy levels across all three school types is quite low the skills required by the EGRA assessment were too difficult for the majority of assessed students. Figure 3 shows the percentage point increase in test scores for the EGRA subscales, including an overall 7 percentage point increase in literacy skills. After 18 months, Impact Network students perform similarly to government schools on the ZAT Pre-Reading Recognition test. Because of the poor levels of performance on the EGRA test, the 18-month follow-up added the Zambian Achievement Test (ZAT). 9 There remains significant room for improvement at all of the school types. Overall levels on the EGRA test are low and indicative that additional work can be done around improving literacy across all schools. The cost per percentage point test score improvement is between 2 and 10 times more expensive at 8

government than at Impact Network schools. For instance, improving the overall EGMA score by one percentage point would cost approximately 3,500 ZMK at an impact school, but more than 30,000 ZMK at a government school. Not only does the eschool 360 program improve the performance of students, particularly in numeracy and some literacy skills, but it also does so in a more cost efficient way than the established schooling institutions. NEXT STEPS Impact Network is using the results of this evaluation to document the eschool 360 model and share insights with policymakers in Zambia. Additionally, they have recently established a partnership with American Institutes for Research (AIR) to develop new interventions and strengthen existing programs before expanding the eschool 360 across the country. This expansion will focus on implementing the eschool 360 in 60 additional schools in partnership with the government of Zambia. 9

ENDNOTES 1. UNESCO. (2014). Progress in getting all children to school stalls but some countries show the way forward. Education for All Global Monitoring Report and the UNESCO Institute for Statistics. Retrieved from: http://www.uis.unesco. org/factsheets/documents/fs-28-outof-school-children-en.pdf 2. The Brookings Institute. (2011). Calculations based on data from the 2011 UNESCO Education for All Global Monitoring Report. Center for Universal Education. Retrieved from: http://www. brookings.edu/ research/interactives/ africa-learning-barometer. 3. Basic math is tested with the Early Grade Mathematics Assessment (EGMA) and literacy is tested with the Early Grade Reading Assessment (EGRA). Both instruments were developed by RTI International with the support of the World Bank and USAID, are an internationally recognized tool and freely available online. 4. Schling, M., Winters, P. (2013). Baseline data report for the Impact Network elearning education program in rural Zambia. Report prepared for Impact Network. 5. Schling, M., Winters, P. (2013). Qualitative data collection report for the Impact Network elearning education program in rural Zambia. Report prepared for Impact Network. 6. Using the methodology proposed by Levin and McEwan (2001). 7. Winters, C., Schling, M., Winters, P. (2013). Cost effectiveness report for the Impact Network elearning education program in rural Zambia. Final Report prepared for Impact Network. 8. Schling, M., Winters, P. (2014). Impact evaluation report for the Impact Network elearning education program in rural Zambia. Final Report prepared for Impact Network. Overall, this evaluation reveals impact results that are encouraging, but have to be interpreted with some caution. The EGRA test may have been a potentially inadequate measure of literacy skills at such low levels of performance. Additionally, there were econometric difficulties when trying to ensure bias-free impact estimates. The estimates presented here represent the unweighted impact results, though the results with the weighted results from propensity score matching do not vary significantly from what is presented here. 9. The ZAT was developed for use in multiple Zambian languages, and constructed in the Zambian context to measure academic achievement. For the purposes of the follow-up study, the pre-reading skills test was used. Note that the ZAT score is only available for follow-up, so all students in Impact Network schools have received some treatment by that time. 10