Asian Development Bank - International Initiative for Impact Evaluation. Video Lecture Series

Asian Development Bank - International Initiative for Impact Evaluation Video Lecture Series Impact evaluations of social protection- Project and Programmes: considering cash transfers and educational outcomes

Overview Social protection: transfers, pensions, and public works Cash transfers and educational outcomes: theory of change Builds on Video Lecture 2 Example: Scholarship Programme (CSP) in Cambodia Research questions Regression discontinuity (RD) design Builds on Video Lecture 4 Systematic review: Balance of evidence of effect of cash transfers on education outcomes Builds on Video Lecture 5 MARK EDWARDS / Still Pictures

Overview Social protection: transfers, pensions, and public works Cash transfers and educational outcomes: theory of change Builds on Video Lecture 2 Impact evaluation example: Cambodia Education Sector Support Project Scholarship Programme (CSP) Research questions Regression discontinuity design (RDD) Builds on Video Lecture 4 Systematic review: Balance of evidence of effect of cash transfers on education outcomes Builds on Video Lecture 5

Conditional cash transfers (CCTs) Variants of the idea are ~200 years old but have gained particular attention in the past 15 years 1997: publication of evaluation of PROGRESA (later, Opportunidades) programme, in Mexico https Particularly ://fbc dn-sphotos-h-a.akamaihd.net/hphotos-ak-xpf1/t31.0-8/c0.208.851.3 spread through 15/p851x315/10293742_730728400306315_1813145547174495948_o.jpg Latin America, with later efforts in Africa and Asia Now in effect in >37 L&MICs In Asia: Bangladesh Cambodia Indonesia Philippines Turkey Foundation Escalera One of the most evaluated social programmes

Conditional cash transfers (CCTs) Variants of the idea are ~200 years old but have gained particular attention in the past 15 years 1997: PROGRESA (later, Opportunidades) programme, in Mexico Particularly spread through Latin America, with later efforts in Africa and Asia https ://fbc dn-sphotos-h-a.akamaihd.net/hphotos-ak-xpf1/t31.0-8/c0.208.851.3 15/p851x315/10293742_730728400306315_1813145547174495948_o.jpg Now in effect in >37 L&MICs In Asia: Bangladesh Cambodia Indonesia Philippines Turkey One of the most evaluated social programmes Foundation Escalera

Conditional cash transfers (CCTs) Variants of the idea are ~200 years old but have gained particular attention in the past 15 years 1997: PROGRESA (later, Opportunidades) programme, in Mexico Particularly spread through Latin America, with later efforts in Africa and Asia Now in effect in 37+ low- and middle-income countries In Asia: Bangladesh Cambodia Indonesia Philippines Turkey It is fair to say that CCTs are one of the most studied programmes in development economics. It is fair to say that CCTs are one of the most studied programmes in development economics. Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

UCTs v CCTs: current debate Unconditional cash transfer (UCT) \households people lack money, not rational priorities for how to use or invest it to suit their own needs Conditional cash transfer (CCT) Market failure related schooling for children Lack of information on true costs and value Induce change through changing relative price of schooling Not just income effect of cash transfer Political feasibility May be more palatable to taxpayers (and donors) if transfers are seen as promoting/rewarding socially desirable, rather than being handouts behavior Capt. John Severns, U.S. Air Force http://upload.wikimedia.org/wikipedia/commons/9/99/schoolgirls_in_bamozai.jpg Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

UCTs v CCTs: current debate Unconditional cash transfer (UCT) Households lack money, not rational priorities for how to use or invest it to suit their own needs Conditional cash transfer (CCT) Market failure related schooling for children Lack of information on true costs and value Induce change through changing relative price of schooling Not just income effect of cash transfer Political feasibility May be more palatable to taxpayers (and donors) if transfers are seen as promoting/rewarding socially desirable, rather than being handouts behavior Capt. John Severns, U.S. Air Force http://upload.wikimedia.org/wikipedia/commons/9/99/schoolgirls_in_bamozai.jpg Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

UCTs v CCTs: current debate Unconditional cash transfer (UCT) Households lack money, not rational priorities for how to use or invest it to suit their own needs Conditional cash transfer (CCT) Market failure related schooling for children Lack of information on true costs and value Induce change through changing relative price of schooling Not just income effect of cash transfer Political feasibility May be more palatable to taxpayers (and donors) if transfers are seen as promoting/rewarding socially desirable, rather than being handouts behavior Capt. John Severns, U.S. Air Force Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Theory of change: transfers Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Theory of change: transfers Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) Teacher attendance, skill, effort Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Theory of change: transfers Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) School fees (relative to size of transfer and household income) Teacher attendance, skill, effort Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Theory of change: transfers conditionality Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) School fees (relative to size of transfer and household income) Teacher attendance, skill, effort Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Impact Evaluation (IE) example Key research question: What is the causal (attributable) impact of being offered a cash transfer on: enrolment Attendance Test scores (mathematics and vocabulary) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Impact Evaluation (IE) example Key research question: What is the causal (attributable) impact of being offered a cash transfer on: Enrolment Attendance Test scores (mathematics and vocabulary) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

IE example: CSP in Cambodia Scholarships (transfers) to low-income households for the lower secondary school cycle (3 years; Grade 7, 8, 9) Scholarships paid out three times per year, conditional on: enrolment Attendance Grade progress Amount of scholarship was approximately the direct cost of secondary school Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

IE example: CSP in Cambodia Scholarships (transfers) to low-income households for the lower secondary school cycle (3 years; Grade 7, 8, 9) Transfers paid out three times per year, conditional on: Enrolment Attendance Grade progress Amount of scholarship was approximately the direct cost of secondary school Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

IE example: CSP in Cambodia Scholarships (transfers) to low-income households for the lower secondary school cycle (3 years; Grade 7, 8, 9) Transfers paid out three times per year, conditional on: Enrolment Attendance Grade progress Amount of transfer approximately the cost of secondary school fees (direct costs) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Selection process Selection of lower secondary schools Served low-income areas High levels of non-enrolment, drop-out Collection of primary school data Assignment of a drop-out risk score Eligibility Enrolling in school Attending school Academic achievement (test scores) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Selection process Selection of lower secondary schools Collection of primary school data Assignment of a drop-out risk score All 6 th grade students potentially attending that secondary school filled out an application: gender, household characteristics All 6 th grade students across the country completed the application on the same day Eligibility Enrolling in school Attending school Academic achievement (test scores) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Selection process http://upload.wikimedia.org/w ikipedia/commons/1/16/school_children_%28cambodia%29.j peg Selection of lower secondary schools Collection of primary school data Assignment of a drop-out risk score Eligibility Natasha Graham, EFA FTI Score created from student and household factors that predicted likelihood of not enrolling in 7 th grade, having completed 6 th grade in national data In schools with: Enrolling in school Attending school Academic achievement (test scores) > 200 students, bottom 50 students offered 200 students, bottom 30 students offered This creates a discontinuity Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Evaluation design (RD) Selection of lower secondary schools Collection of primary school data Assignment of a drop-out risk score Eligibility Enrolling in school Attending school Academic achievement (test scores) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Evaluation design (RDD) An eligibility criterion can be used in a regression discontinuity design for evaluating causal impact. One of the quasi-experimental methods Recall that: Attributing causal impact to an intervention requires a comparison group Approximate what would have happened in absence of the treatment One way to construct a comparison group uses knowledge of a well-understood, programme-imposed cut-off for inclusion This cut-off creates a discontinuity Dee_/Flickr

Evaluation design (RDD) Hypothetical large school, School A

Evaluation design (RDD) 50 46 39 38 37 35 35 33 32 14 11 Dropout risk score

Evaluation design (RDD) Offered the transfer (50 most at-risk students) Eligibility cutoff 50 46 39 38 37 35 35 33 32 14 11 Dropout risk score

Evaluation design (RD) Comparable? 50 46 39 38 37 35 35 33 32 14 11 Dropout risk score

Evaluation design (RD) Assumption for evaluating impact: these groups are similar pre-treatment 50 46 39 38 37 35 35 33 32 14 11 Dropout risk score Note that: This assumption is partially empirically testable with baseline data But, assumption also rests on similarity of unobservables This assumption depends on an arbitrary cut-off

Proportion of students attending school Evaluation results Relative ranking of dropout risk (line of best fit) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Proportion of students attending school Evaluation results Eligibility cut-off Eligible Not eligible (Comparison group) Relative ranking of dropout risk (line of best fit) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Proportion of students attending school Evaluation results Eligibility cut-off Eligible Not eligible (Comparison group) Distinct jumps at the cut-off would suggest that the programme affected outcomes. Relative ranking of dropout risk (line of best fit) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Proportion of students attending school Evaluation results Eligibility cut-off Eligible Not eligible (Comparison group) ~25 percentage point change in attendance (sample size: 3225) Relative ranking of dropout risk (line of best fit) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Test scores (normalised; for vocabulary) Evaluation results Eligibility cut-off Eligible Not eligible (Comparison group) Relative ranking of dropout risk (line of best fit) Filmer, D. and N. Schady. School enrolment, Selection and Test Scores. Impact Evaluation Series 34. Policy Research Working Paper 4998. The World Bank. 2009: 7. http://www.worldbank.org/en/news/video/2014/04/15/cambodias-45-scholarships-what-is-the-impact

Balance of evidence from SR What does a systematic review (SR) tell us about the impacts of being eligible for a cash transfer on schooling achievement? inclusion This cut-off creates a discontinuity Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Line of no effect Balance of evidence from SR CCTs on test scores Programme name Country Standard deviation change (95% CI) CESSP Scholarship Programme Cambodia Systematic reviews often show Tayssir Morocco evidence using a forest plot Nahouri Cash Transf er Pilot Burkina Faso -0.04 (-0.23, 0.15) 0.01 (-0.06, 0.08) 0.11 (-0.06, 0.27) SIHR Malawi 0.13 (0.02, 0.24) Red de Proteccion Social Nicaragua 0.21 (0.05, 0.37) Total (l-squared = 50.9%, p = 0.086) 0.08 (-0.00, 0.16) Intervention reduces test scores Intervention increases test scores Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Balance of evidence from SR CCTs on test scores Programme name Country Standard deviation change (95% CI) CESSP Scholarship Programme Cambodia -0.04 (-0.23, 0.15) Tayssir Morocco 0.01 (-0.06, 0.08) Nahouri Cash Transf er Pilot Burkina Faso 0.11 (-0.06, 0.27) SIHR Malawi 0.13 (0.02, 0.24) Red de Proteccion Social Nicaragua 0.21 (0.05, 0.37) Total (l-squared = 50.9%, p = 0.086) 0.08 (-0.00, 0.16) Intervention reduces test scores Intervention increases test scores Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Balance of evidence from SR CCTs on test scores Programme name Country Standard deviation change (95% CI) Cambodia CESSP Scholarship Programme Programme Cambodia -0.04 (-0.23, 0.15) 0.15) Tayssir Nahouri Cash Transfer Pilot SIHR Red de Proteccion Social Morocco Burkina Faso Malawi Nicaragua 0.01 (-0.06, 0.08) 0.11 (-0.06, 0.27) 0.13 (0.02, 0.24) 0.21 (0.05, 0.37) Intervention reduces test scores Intervention increases test scores Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Balance of evidence from SR CCTs on test scores Programme name Country Standard deviation change (95% CI) Cambodia CESSP Scholarship Programme Programme Cambodia -0.04 (-0.23, 0.15) 0.15) Tayssir Nahouri Cash Transfer Pilot SIHR Red de Proteccion Social Morocco Burkina Faso Malawi Nicaragua 0.01 (-0.06, 0.08) 0.11 (-0.06, 0.27) 0.13 (0.02, 0.24) 0.21 (0.05, 0.37) Total l-squared = 50.9%, p = 0.086) Cumulative (pooled) effect size 0.08 (-0.00, 0.16) Intervention reduces test scores Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 Intervention increases test scores No effect of being offered CCTs on learning outcomes.

In sum Theory of change that cash transfers may improve achievement by encouraging school enrolment and attendance. In Cambodia, a CCT did increase enrolment and attendance but not test scores. In a systematic review, see similar results It is fair to say that CCTs are one of the most studied programmes in development economics. Still a lot to learn in social protection! Natasha Graham, EFA FTI

In sum Theory of change that cash transfers may improve achievement by encouraging school enrolment and attendance. In Cambodia, a CCT did increase enrolment and attendance but not test scores. In a systematic review, see similar results. It is fair to say that CCTs are one of the most studied programmes in development economics. Still a lot to learn in social protection! Natasha Graham, EFA FTI

Theory of change sh sfer Income Enrolling in school Attending school Academic achievement (test scores) Skills attained

Theory of change Enrolling in school Attending school Academic achievement (test scores) Skills attained Labour market outcomes

Theory of change Attending school Academic achievement (test scores) Skills attained Labour market outcomes Welfare

Theory of change Eligibility Cash transfer Income Enrolling in school Attending school Academic achievement (test scores) Enforcement of the condition Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Balance of evidence (scatter plot) Transfers, enforcement, and school attendance UCT Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8

Balance of evidence from SR Transfers, enforcement, and school attendance UCT Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 CCT

Balance of evidence from SR Transfers, enforcement, and school attendance UCT Increasing enforcement of conditions Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 CCT

Likelihood (odds) of attending school Balance of evidence from SR Transfers, enforcement, and school attendance UCT Increasing enforcement of conditions Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 CCT

Likelihood (odds) of attending school Balance of evidence from SR Transfers, enforcement, and school attendance CSP Cambodia UCT Increasing enforcement of conditions Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 CCT

Likelihood (odds) of attending school Balance of evidence from SR Transfers, enforcement, and school attendance Line of best fit UCT Increasing enforcement of conditions Baird, S., Ferreira, F. H. G., Özler, B., Woolcock, M. Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review. Campbell Systematic Reviews 2013:8 CCT