Evaluating the impact of an education programme

Transcription

1 Asian Development Bank - International Initiative for Impact Evaluation Video Lecture Series Evaluating the impact of an education programme

2 Making education count Significant progress in meeting this Millennium Development Goal Ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling. But has enrolment led to learning?

3 Education interventions Supply side interventions Demand side interventions Photo school-buiding/openclipart,books/pixabay,guilia forsythe/flickr,cash transfer/wikimedia,scholarship/openclipart,school feeding/rockinteachermaterials

4 Outcomes Many of these interventions have improved: Enrolment Attendance Grade progression But there is less evidence of what works in increasing learning. Photo 3ie wp_20

5 Case of India School enrolments are now over 95% in India. But only 47% of students in standard 5 can read standard 2 level text. Only 76% of standard 8 students can read standard 2 level text. Photo Pratham books

6 Enhancing learning in India This 3ie supported impact evaluation was conducted in Mahendragarh and Kurukshetra districts of Haryana, India. Impact evaluation of two programmes: Continuous and Comprehensive Evaluation system (CCE) Learning Enhancement Programme (LEP) Researchers from J-PAL collaborated with the Government of Haryana. Photo Haryana_in_India/wikimedia Duflo, E, Berry, J, Mukerji, S and Shotland, M, A Wide Angle View of Learning: evaluation of the CCE and LEP Programmes in Haryana, 3ie Impact Evaluation Report

7 Continuous Comprehensive Evaluation The Right to Education Act (2009) eliminated high stake final exams. The Continuous and Comprehensive Evaluation system emphasises: Photo Yann/wikimedia Frequent and broad based feedback on student performance Assessment of academic and non-academic performance Variety of techniques to assess performance Photo Chris hand/flickr

8 Learning Enhancement Programme Developed by Pratham, a large Indian NGO focussing on basic literacy and numeracy Learning Programme involves quick oral assessment of students Classes are regrouped according to learning level rather than grade. Paragraph Word Letter Story Beginner Photo Devanagari_velars/wikimedia,Bhasha/wikimedia,Hindi_Sahitya_puraskar/wikimedia

9 Process monitoring On the request of the government, researchers revived school monitoring system. The system includes block and district supervisors, field level monitors. Monitoring consisted of Surprise visits to each of the schools Extensive questionnaires on implementation, availability of inputs such as text books and uniforms Photo Ray witlin/wb Observations of randomly selected teacher in the classroom

10 Research questions Does CCE improve test scores in Hindi and Maths? Does LEP improve test scores in Hindi and Maths? Does a combination of CCE and LEP improve test scores in Hindi and Maths? Photo Counterclockwise/Flickr,Magic Pathshala.com

11 Randomised Controlled Trial 400 schools 100 CCE schools 100 LEP schools 100 CCE+LEP schools 100 schools in control group A cluster randomised controlled trial with a factorial design Grades 1 to 4

12 Impact evaluation method Unit of assignment Unit of treatment Unit of analysis School campus School Student Sample and data Primary source of data was test scores in Hindi and Maths Tests were administered to up to 10 randomly selected students in each grade Photo Kristian Niemi/Flickr Timeline : Baseline testing Photo Room to Read : Implementation of the programme and endline testing Total sample of 12,576 students Surveys of head masters for teaching and evaluation practices

13 Results Students in CCE schools did not perform significantly better than students in control Schools. Photo Jose Antonio Marcillo Valenciano/Flickr LEP had a large, positive and statistically significant effect on students basic Hindi reading abilities, both oral and written tests. LEP had a larger effect for girls than boys. Photo Haryana students/jpal SA Combining LEP and CCE had no significant effect on test scores relative to the LEP programme alone. Photo Jose Antonio Marcillo Valenciano/Flickr

14 Why didn t CCE work? Process monitoring showed: CCE training did not lead to change in teaching practices. School headmasters thought requirements to be burdensome and time consuming. Guidelines were unclear Overall, CCE not well implemented. LEP on the other hand had high level of compliance and was well implemented

15 CCE Theory of Change CCE Teacher training Frequent evaluation of students Customised teaching approach Improvement in learning Teacher training is adequate. Teachers have time and resources to implement. Teachers have pedagogical tools. Teachers attend training Assumptions not met

16 LEP Theory of Change LEP Teacher training Evaluate and restructure class Teach at the appropriate level Improvement in learning Teacher training is adequate. Teachers have time and resources to implement. Teachers have pedagogical tools. Teachers attend training Assumptions met in most part

17 Policy implications and lessons Government of Haryana has now commissioned a detailed review of CCE. Photo Michael Foley/Flickr Pratham has used the evaluation to build partnerships with other state governments. Photo collab-hands/cirtblog Ongoing and deep engagement of researchers with the government helped in uptake of findings. Photo AM NET/Japan

18 In sum The impact evaluation highlights how teaching that is customised to learning ability can help in improving learning outcomes. Photo Alfredo Srur/WB The importance of training teachers in the use of pedagogical tools. Process monitoring is crucial for ensuring programmes are well-implemented. Constant and deep engagement of researchers with policymakers helps in uptake of research.