INTEGRATED EDUCATION PROGRAM (IEP) Report: Review of the Impact of the Integrated Education Program (IEP)

Transcription

1 INTEGRATED EDUCATION PROGRAM (IEP) Report: Review of the Impact of the Integrated Education Program (IEP) July 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Eric Schollar for RTI International.

2 INTEGRATED EDUCATION PROGRAM (IEP) REPORT: REVIEW OF THE IMPACT OF THE INTEGRATED EDUCATION PROGRAM (IEP) USAID Contract No. 674-C Prepared by RTI International for USAID/Southern Africa RTI International 3040 Cornwallis Road Post Office Box Research Triangle Park, NC USA

3 CONTENTS Background and Context Objectives and Methods. 2 Data Sources for Impact on Learner Performance... 3 Impact on Learner Performance Cohort 1 to Cohort 4a 7 Cohort 4b... 9 Conclusions 11 Investigating the Nature and Distribution of Program Impact The Causes of Impact on Learner Performance. 16 Why did Impact Occur in Grade 3 only in Cohort 1-3 and Cohort 4a? Some Key Issues in Intervention Delivery that Affect Learner Performance Involvement of Local-level DoE Officials 19 Master Teachers Allocated out of the Program. 20 Cascade Training Classroom support. 21 Result Areas and Impact on Learner Performance Result Result Result Result Result Result Result Conclusion. 28 Acronyms ARB Assessment Resource Bank CASS Continuous Assessment DDSP District Development Support Program DoE Department of Education LCD Link Community Development TVT The Valley Trust NCS National Curriculum Statement NIHE National Institute for Higher Education NSE National Systemic Evaluation

4 Review of the Impact of the IEP 1 1. Background and Context The Integrated Education Program (IEP) was a primary education improvement program that operated in the KwaZulu-Natal, Limpopo, Eastern and Northern Cape provinces between 2004 and 2008 funded by $23 million from the U. S. Agency for International Development (USAID). The program was implemented in partnership with the South African Department of Education (DoE) and was a follow-on project to the District Development Support Project (DDSP: ). The IEP incorporated the best practices of the DDSP, including needs-based training directly aligned to the objectives of the DoE. The main objective of the IEP, as that of the DDSP, was the improvement of learner performance. The means through which this was to be achieved were based on systemic strengthening. The strategic causal chain of the IEP can be distinguished as: Improved performance of learners can be achieved through the improvement of the quality of education they receive Quality of education can be improved by supporting existing programs in school management and governance, and in teacher education in selected subjects (literacy, numeracy, mathematics and science). Improved management and teacher capacity will result in improved levels of curriculum management and delivery, at a higher level of pedagogic and content quality. Improved curriculum management and delivery, along with improved classroom instruction, will result in improved learner performance in the four subjects. The operational approach of the IEP is described in nine result areas that are intended to achieve the overarching objective of improved learner performance. Result 1 Result 2 Result 3* Result 4 Result 5 Result 6 Result 7 Result 8 Result 9* Increased subject matter knowledge for teachers in the targeted subject areas Improved ability of teachers to develop and apply continuous assessment strategies and techniques Increased number of teachers being trained in service and new teachers trained Increased number of teachers that are teaching literacy, numeracy, mathematics and science in a manner consistent with the NCS Improved teachers ability to develop and use classroom materials that support learner-centered instruction Improved instructional leadership skills for Scholl Management Teams (SMT) Improved management and administrative capacity of schools to collaborate with School Governing Bodies (SGB) and communities to effectively implement 3-year School Development Plans to improve school functionality Improved district capacity to develop and effectively implement a strategic plan for school support Support to National Department of Education *It should be noted that neither Result 3 nor 9 are dealt with in this review The IEP has a two-tiered management structure. The first layer includes the prime contractor, RTI International, and three core sub-contractors; Centre for Education Policy Development (CEPD), JET Education Services (JET) and Khulisa Management Services (KMS). The second layer comprises the subcontractors and universities appointed to implement the program in the different provinces.

5 Review of the Impact of the IEP 2 Table 1: Implementation structure of the IEP Eastern Cape KwaZulu-Natal Limpopo Northern Cape Co-ordination READ LCD MOLTENO LCD Literacy READ READ MOLTENO MOLTENO Numeracy/Mathematics COUNT COUNT COUNT COUNT Science PROTEC PROTEC PROTEC PROTEC EMGD LCD LCD LCD LCD HIV/AIDS TVT TVT Akanani TVT Pre-service UFort Hare UZululand ULimpopo UVenda NIHE There were a total of 1,284 schools involved in the program: Eastern Cape 306 (24%) KwaZulu-Natal 333 (26%) Limpopo 466 (36%) Northern Cape 179 (14%) These schools were divided into three cohort groups, each entering the program at a different time over the 2004 to 2007 period. Cohort 1 to 3, comprised schools from all of the provinces 1 and entered the IEP program over the 2004/2005 period. Cohort 1 and 2 schools were former DDSP schools, while Cohort 3 schools were a new group of schools. Cohort 4a, new schools in Limpopo, entered in late Cohort 4b, new schools in all of the provinces except Limpopo, entered in the 2006 to 2007 period. The IEP teacher training process consisted of three approaches implemented in stages: Residential training provided by IEP trainers for Master Teachers (MT) held during school holidays Cluster workshops provided by MTs and IEP trainers for cascadee teachers held after hours during school terms School-based support provided by IEP trainers to both MTs and cascadees during the school day. 2. Objective and Method The review is essentially a qualitative assessment of the nature and causes of the impact achieved by the IEP and, therefore, takes as its organizing theme the main objective of the program impact on learner performance along with a discussion of the different result areas in relation to the achievement, or otherwise, of this objective. In essence, the learner impact data has been extensively scrutinized for patterns that suggest where impact was greatest, where qualitative investigations should be targeted and what the causes of the impact may, or may not, have been. The method of the review was to use the existing data, supported by a range of interviews and provincial visits, to develop a qualitative, sometimes speculative, reflection of what happened to learner performance in the IEP and why. It attempts to relate the practices of the IEP, and the changes it achieved, to changes in learner performance. In doing so, the review is based on the same conceptual divisions as the IEP itself. This approach distinguishes between the: 1 Schools from Limpopo in Cohort 1-3 were dropped by IEP in August 2005.

6 Review of the Impact of the IEP 3 Ends: Improved student performance in numeracy, literacy, mathematics and science for students in participating schools. Means: The IEP supports teachers, education officials and school communities to improve the quality of curriculum delivery and to develop effective school management and governance at the primary school level. JET Education Services was commissioned to carry out the assessment of learner performance and teacher practice as well as the analysis of teacher content knowledge testing, and data from these studies form the fundamental basis of the review. In addition, the review uses data gathered from: Twelve interviews with sub-contractors Nine days of provincial visits during which: o o IEP staff members and sub-contractors were interviewed A number of schools, selected on the basis of their performance in the learner testing, were visited. At each visit, interviews were held with the principal or other SMT members, master and cascade teachers. o District and provincial officials of the DoE were interviewed. Project reports and other documents provided by IEP. The JET studies deal with sustained, detailed and comprehensive quantitative and qualitative processes; readers should refer to these documents for detailed information about performance against formal IEP indicators. In addition, IEP has itself produced a number of documents detailing performance against indicators for all of the provinces; these documents will also provide quantitative and qualitative information in more detail than is reflected in this review. 3. Data Sources for Impact on Learner Performance The impact of the IEP on learner performance is based on a quasi-experimental model for both project and control groups. Figure 1: Conceptual Design of the IEP Evaluation 2 Project Intervention & routine schooling Baseline Post project research research Control Routine schooling only Since both groups continue to receive normal schooling over the life of the project, we can expect there to be some gains in the scores of the project and control groups even if the IEP had never been implemented. Therefore, in order to account for this effect on the project group, impact analysis compares the growth in mean scores between pre- and post-testing achieved by the two groups essentially, the gain of the control group is subtracted from that of the project group to measure the impact of the intervention program. In terms of the reliability of measurements, the first question that must be considered is the sample upon which testing was based. The project group is selected from the target population (i.e. schools participating in the IEP) and must be sufficiently representative of it for reliable conclusions about the performance of the group to be generalized to the whole population. Equally, the selected control schools must also be sufficiently representative of the target population to sustain reliable generalization. 2 Note that a mid-project test was added for Cohorts 1-3 and 4a, and a second post-project test for Cohort 1-3.

7 Review of the Impact of the IEP 4 According to the JET reports, there were a total of 95 schools in the three cohort groups involved in the evaluation, 82 project and 13 control, in Table 2: Provincial distribution of project and control schools by cohort E. Cape KZN N. Cape Limpopo Project Control Project Control Project Control Project Control Cohort Cohort 4a 16 2 Cohort 4b It will be immediately evident that the control groups are very small. The proportions of these groups in relation to the whole sample in each province are: Cohorts % Cohort 4a 11.1% Cohort 4b 15.0% When the project and control groups are of different sizes, the data they yield has correspondingly different sampling errors in terms of their generalization to the whole population. This essentially means that they provide data at different levels of precision. To illustrate, and to arrive at a figure for significance to be used throughout the review, we consider the case of the largest (and longest lived, see below) of the three cohorts. At baseline, there were 637 schools in the Cohort 1-3 target population and testing was conducted at Grades 3 and 6 by randomly selecting 25 learners from each of these grades at each school in the project and control groups. Let us assume that the IEP schools have a minimum of two classes at each grade level and that there is an average of 35 learners in each class. This yields a population of 637 schools x 2 grades x 2 classes x 35 learners, or 89,180 learners in all. The test sample for this cohort was based on 1,542 learners with 1,370 and 172 in the project and control groups, respectively. Using tables for determining sample size against population (Krejcie and Morgan, 1970) 3 these numbers provide data at a precision of around 3.5% for the project group, and 7.5% for the control group, at a confidence level of 95%. This means that we can be 95% confident that mean learner scores derived from the two groups is accurate to within these levels of precision. Since we need to determine a convenient, reasonable and consistent value for the level of significance that will be used in the review, and since this is not a scientific paper, I have somewhat arbitrarily chosen 4% for all of the cohorts. This implies that measured impacts, and especially those based on project/control comparisons, need to be, at least, above +4% or below -4% before they can be considered to be significant in providing suggestive evidence of intervention effect, positive or negative. The larger the impact figure over 4%, the more likely that it is significant and indicates intervention effect rather than sampling error or other random variables. Nonetheless, it should still be remembered that the impact figures discussed in this review are more likely to be reasonably indicative, rather than exact, measures of impact on learner performance. In this regard, it should also be noted that each cohort was tested a different number of times. Cohort 1-3 was post-tested twice, once immediately after the end of the program and again one year later. This is an excellent practice and provides two related benefits. The first is the ability to discern longer term trends in impact scores and the second is the ability to investigate the issue of sustainability of intervention effect after end-project. Unfortunately, this practice was not possible 3 Kreijie and Morgan (1970) Determining sample sizes for research activities. Education and Psychological Measurement, pp

8 Review of the Impact of the IEP 5 for Cohorts 4a or 4b, although Cohort 4a was tested three times (pre, mid and post) and so does still provide trend data. Cohort 4b was tested only twice (pre and post) and, consequently, provides a linear measure of impact without any indication of trend. Table 3: Test process Baseline Mid Post 1 Post 2 Cohort Cohort 4a n/a Cohort 4b n/a n/a As the process implies, the test process and duration was dictated by the date of entry to the IEP of the different cohorts. Cohort 1-3 was involved with the IEP for the longest period and Cohort 1-2 was also in the DDSP. Cohort 4b, by contrast, had only just over one year s exposure and none of the schools were involved in the DDSP. Therefore, the focus of the analysis that follows is most often placed on Cohort 1-3, though the others are also considered in some detail. Secondly, when comparing the relative performance of cohorts, the second post-test scores are often discarded for Cohort 1 3 for the sake of comparability with both Cohorts 4a and 4b which did not have a similar test process. 4. Impact on Learner Performance 4.1. Cohort 1-3 As we have already noted, Cohort 1-3 was involved with the IEP over the longest period, included schools which had been in the DDSP and data was taken at four measurement points. It is, consequently, prima facie the cohort most likely to provide reliable information about the impact of the IEP. Table 4: Cohorts 1 to 3: Project/Control Impact Analysis: All Provinces Grade 3 Grade 6 Literacy Numeracy Combined Math Science Combined Project Baseline Mid Change: baseline to mid Post Change: mid to post Post Change: post 1 to post Gain: baseline to post Control Baseline Mid Change: baseline to mid Post Change: mid to post Post Change: post 1 to post Gain: baseline to post Impact: baseline to post

9 Review of the Impact of the IEP 6 Figure 2: Cohort 1-3: Summary of Impact Impact Literacy Numeracy Grade 3 Math Science Grade 6 Test It is very clear that the impact of the IEP in Cohort 1 to 3 occurred exclusively at Grade 3 level the Grade 6 figures are not significant. Secondly, it will be recalled that impact data is likely to be affected by the different sampling errors of the project and control groups in all of the cohorts; consequently the precision of the data the point at which it becomes significant - is, at least, around 4%. On this reading, only the impact data for literacy can be considered significant and we conclude that: Impact occurs at Grade 3 level only At this level, impact at Grade 3 level is significant for literacy only. However, it should be noted that the conclusion concerning the subject impacts are reversed if we measure the impact from baseline to Post 1, rather than Post 2 tests. A comparative table shows that the gain scores for these subjects for the project and control groups between Post 1 and 2 behaved rather strangely in relation to each other over first post-project period. Table 5: Difference in score for subjects Literacy Numeracy Mid to Post 1 Post 1 to 2 Mid to Post 1 Post 1 to 2 Project Control One would usually seek an explanation in the ending of intervention effects in the period after the end of the project (i.e. these effects are not sustained in the project group) but these figures make that a little problematic. Firstly, the project scores for literacy do indeed decrease over this period but remain well above their baseline, while the control scores decline even further to below their baseline, resulting in a net impact in the project group by Post 2 that was not there at Post 1. On the other hand, the project numeracy score actually increases slightly over this period to well above baseline while the control scores increase much more dramatically between Post 1 and Post 2, resulting in a net loss of impact that was there at Post 1. These patterns can be most easily seen in graphic format.

10 Review of the Impact of the IEP 7 Figure 3: Trends in Scores: Cohort Mean Score Project Literacy Control Literacy Project Numeracy Control Numeracy 15 Baseline Mid Post 1 Post 2 Test The remarkably close symmetry of the literacy scores for both groups up to Post 1 suggest that both of them were strongly affected by external factors in their shared educational context - with some evidence of intervention effect in the project group - that suddenly stops acting over the period between Post 1 and Post 2 after project-end. This general decline in both groups is slowed by, presumably, sustained intervention effect, resulting in a significant impact measurement by Post 2. On the other hand, the numeracy scores behave very differently; after a similarly general decline between baseline and mid, the intervention appears to take effect and the project scores increase strongly while the controls remain almost static between Mid and Post 1. Then the intervention ends and the project score slows its growth, as might be expected, but the control matches the earlier increase of the project, which would not be expected, resulting in no significant impact since baseline. On the face of it, the speculative conclusions that one can reach about the post-project period are that: In literacy, there is a general decline in learner performance that is slowed by the effect of the IEP. This is encouraging in terms of evidence for a sustained intervention effect but not quite so positive in that the trend is negative. In numeracy, there is an unexplained general improvement underway in the control group that was possibly even blunted by the effects if the IEP in the project group i.e. why should they not also respond to whatever was affecting the controls? Nonetheless, it is still evident that the trend for the project score is still positive, if slower than prior to project-end, and that suggests evidence for sustained intervention effect. One way or the other, it is curious that the pattern of significant impact was reversed in the period after the end of the project Cohort 4a It will be recalled that this cohort was tested three times - baseline, mid-project and at end-project. Although it had the lowest numbers of schools from which the sample was drawn, they were all in one province Limpopo thus reducing variation due to provincial differences. However, it also had the smallest control group % of the sample and has, therefore, the lowest level of precision.

11 Review of the Impact of the IEP 8 Table 6: Cohort 4a: Project/Control Impact Analysis Grade 3 Grade 6 Literacy Numeracy Combined Math Science Combined Project Baseline Mid Change: baseline to mid Post Change: mid to post Gain: baseline to post Control Baseline Mid Change: baseline to mid Post Change: mid to post Gain: baseline to post Impact: baseline to post Figure 4: Cohort 4a: Summary of Impact Impact Literacy Numeracy Grade 3 Math Science Grade 6 Test The most obvious feature of the data is the confirmation of the conclusion reached with regard to Cohort 1-3; the impact of the IEP occurred exclusively at Grade 3 level and the measured impacts for Grade 6 are not significant. Unlike Cohort 1-3, impacts on literacy and numeracy are both large enough to be significant as measures of intervention effect though it will be noted that the magnitude of this impact is increased by negative gains in three of the four measures in the control group. Cohort 1-3 impact figures for Grade 3 were also influenced by negative gains in the control group but to a lesser degree in that three of the six measures for the controls recorded negative gains. Overall, it is difficult to ignore the suggestion that there is an unacceptable degree of negative growth within the routine system represented by the controls that, in turn, significantly increases the positive impacts recorded for the project groups which obtained absolute gains of +16% and +11% for literacy and numeracy, respectively. Six of the nine measures (66.7%) indicate declining learner performance in the routine system that is slowed, sometimes reversed, by the IEP; one would

12 Review of the Impact of the IEP 9 expect, if anything, positive gains in the controls over time albeit lower than those recorded by the project schools that receive the added benefit of an external intervention programme. Figure 5: Trends in Scores: Cohort 4a Mean Score Baseline Mid Post Test Project Literacy Control Literacy Project Numeracy Control Numeracy It is evident that all of the growth in the scores of the project group occurred in the period immediately after intervention delivery, reversing the trend displayed by Cohort 1-3 in which scores declined immediately after delivery and in which virtually all of the growth occurred in the period between Mid and Post 1 testing when scores for Cohort 4a showed no growth at all (literacy scores actually declined). The only intervention difference between the two cohort groups of which I am aware, and which is discussed at greater length in the following sections of this review, is that learner workbooks and assessment materials were provided to Cohort 4a in 2007 (the Mid to Post 1 period), whereas only KwaZulu-Natal in Cohort 1-3 schools received similar support in 2006 (Mid to Post 1). This helps to explain why the overall impact is so much higher in Cohort 4a. However, it does not explain why the project schools in Cohort 4a achieved no/negative growth in the period between mid and post-testing, though it is interesting to note that project scores declined in both cohort groups in the period immediately after they had just achieved their largest improvements. The obvious suggestions are that intervention effects: Are quick-acting once improvement starts with the supply of materials Are very short lived, slowing, flattening or even declining immediately after they are achieved Cohort 4b Like Cohort 1-3, Cohort 4b was implemented in KwaZulu-Natal, Eastern Cape and Northern Cape but consisted entirely of new schools that had not participated previously in the DDSP. Of the three cohort groups, it enjoyed the shortest exposure to the IEP program although it was the cohort with the largest proportion of the sample represented by the control group (15.0%). Though this is not much higher than the others, it is still inherently the most reliable of the controls for the three cohorts and, consequently, it is something of a pity that it could only be tested twice - making the observation of trends impossible - and then over a very short period of one school year.

13 Review of the Impact of the IEP 10 Table 7: Cohort 4b: Project/Control Impact Analysis Grade 3 Grade 6 Project Literacy Numeracy Combined Math Science Combined Baseline Post Gain: baseline to post Control Baseline Post Gain: baseline to post Impact: baseline to post Figure 6: Cohort 4b: Summary of Impact Impact Literacy Numeracy Grade 3 Math Science Grade 6 Test Firstly, it will be noted that the controls recorded negative gains on both of the Grade 3 measures. Recalling the discussion above, this means that eight of the eleven (72.7%) control scores for all three cohorts at this level record a decline in learner performance, adding extra support to the suggestion that there is a generalized negative trend in the routine system represented by the control schools, that is slowed by the IEP program. Interestingly, however, Cohort 4b appears to have achieved positive impact at Grade 6 level, the only one to do so over this period, while impact at Grade 3 level is concentrated in literacy because of a larger negative gain in the controls. However, it is very difficult to compare these impact figures with those obtained by the other two cohorts because of the short period covered by Cohort 4b figures. The only sensible way to compare the cohorts is to compare the gains of the project groups in each of them at the same stage of intervention delivery of one school-year.

14 Review of the Impact of the IEP 11 Figure 7: Comparison of Project Gain in year 1: All Cohorts Gain in score Literacy Numeracy Grade 3 Math Science Grade 6 Test 7 Cohort 1-3 Cohort 4a Cohort 4b Over this period, it is clear that the big relative differences are the much better performance of Cohort 4a at Grade 3 level, and the better performance of Cohort 4b at Grade 6 level The only negative gain figures over this initial period of intervention delivery are obtained by Cohort 1-3. One could speculate that since Cohort 4a was operated in only one province, there were less contextual and programme variables affecting its scores and, consequently, positive intervention effects were spread more evenly through the whole project group unlike Cohort 1-3 in which they were concentrated in only one part of the project group (in KwaZulu-Natal). However, I can think of no reasonable explanation for the unusual impact (uncertain as it may be) at Grade 6 level in Cohort 4b Conclusions Overall, I believe that the data for Cohorts 1-3 and 4a, obtained over a longer period, provide the most reliable basis for three main conclusions about impact on learner performance. The impact on learner performance was strongly concentrated at Grade 3 level. Only Cohort 4b provided any evidence of impact on Grade 6 but only over a very short period after which we have no idea of the trend in these scores. At Grade 3 level, while it is a moot point (depending on which post-test we use) whether scores for literacy or numeracy produced the strongest evidence of impact in Cohort 1-3, one of them was significant. In addition, the scores for both subjects are significant evidence of impact in Cohort 4a while evidence for impact is strongest for literacy in Cohort 4b. There is support in the patterns in the data for the suggestion that impact was influenced by the supply of materials, and possibly other forms of support, to the project groups at different times and on different scales Comparison of Impact in Cohort 1-3 and DDSP This is a potentially interesting comparison in that the IEP is a planned successor to the DDSP and consciously applied processes in the IEP that were learned, or confirmed, through the DDSP. In addition, the majority of Cohort 1-3 schools were in the DDSP, a mix of the better and worse performing, and IEP schools were tested the same number of times as were the DDSP schools. A pattern remarkably similar to that of the Grade 3 Cohort 1-3 scores was obtained in the DDSP over a similar period of four tests.

15 Review of the Impact of the IEP 12 Figure 8: Trends in Scores: IEP (Cohort 1-3) & DDSP Mean Score Literacy IEP Numeracy IEP Literacy DDSP Numeracy DDSP 15 Baseline Test 1 Test 2 Test 3 Test Besides the evident near-symmetry of the curves, it really striking how virtually all of the gain in mean score occurs in both programs in the second year of program intervention delivery. A couple of other points are also worth noting: The literacy score for the IEP increased rather more sharply than did that of the DDSP. Of the four scores, only that for numeracy in the IEP is still growing, albeit very much more slowly than the year before, and it was doing so some period after the end of the project. While none of the other figures provide any evidence for the maintenance of intervention effect, in terms of maintaining or continuing improvement, the case of numeracy in the IEP provides an interesting exception. While the exercise is probably not strictly valid it is also interesting to consider the scores in a sequence for DDSP and IEP. Figure 9: Trends in Scores: IEP (Cohort 1-3) & DDSP Mean Score DDSP base Test 1 Test 2 DDSP Post Test IEP Base Test 1 IEP Post 1 IEP Post 2 Literacy Numeracy Of course, it is obvious that we are combining two different samples here having little knowledge of the similarity of contextual and other variables operating on them, and are using scores derived

16 Review of the Impact of the IEP 13 from two different instruments. However, the majority of the IEP schools were in the DDSP themselves and I do not think the graphic is completely meaningless, especially in terms of the trends in measured impact. Further, while the literacy baseline scores for the DDSP and the IEP were clearly different, the numeracy scores were virtually equivalent for both groups at baseline. It is obvious how similar is the behaviour of the literacy and numeracy scores, again suggesting that both program interventions had roughly the same levels of impact at the same period of delivery and, quite possibly, for the same reasons. In my opinion, the data clearly supports the earlier speculation based on the behaviour of the gains of the control schools, that intervention programs are being implemented in a context of general declines in learner performance. These declines are reversed in project schools during the most intense periods of intervention delivery when support is at its maximum, and once the program has been internalized to a greater degree by participants. In any case, once the intervention ends, its affects rapidly diminish and the tendency is for scores to trend back toward baseline, given the possible exception we have already noted numeracy in the IEP. 5. Investigating the Nature and Distribution of Program Impact We have already noted in passing that variations in programme delivery are a possible explanation for the patterns in learner performance in the IEP in Cohort 1-3, and there is supportive evidence of this suggestion in the impact data derived from the other cohorts as well. To investigate the matter further, we commence with an analysis of the provincial project group scores in Cohort 1-3, the only cohort that had provincial sub-samples that were tested over a reasonably long period of programme delivery. The figures that are used are those obtained at Post 1, immediately after the project-end, to ensure that they are as comparable as possible with the other cohort groups which did not have a Post 2 test well after end-project Impact at Provincial Level Figure 10: Provincial Impact: Cohort 1-3 Gain between pre and post 1 tests (%) Literacy Numeracy Math Science E. Cape KZN N. Cape Test The impact at Grade 3 level is very strongly concentrated in KwaZulu-Natal, with the addition of numeracy in the Northern Cape.

17 Review of the Impact of the IEP 14 KwaZulu-Natal Literacy Grade % KwaZulu-Natal Numeracy Grade % Northern Cape Numeracy Grade % Figure 11: Trend in Provincial Impact: Grade 3: Cohort Mean Score E. Cape KZN N. Cape 25 Baseline Mid Post 1 Test All three provinces share the whole sample characteristic of virtually all growth occurring after the first year of program delivery this growth is strongest in KwaZulu-Natal. We already know that the sample means for Grade 6 were not significant and, hence, have received virtually no attention to this point. However, the trends in these figures are also interesting. Figure 12: Trend in Provincial impact: Grade 6: Cohort Mean Score E. Cape KZN N. Cape 15 Baseline Mid Post 1 Test Firstly, the scores at the two grades reverse their tendency for both the Eastern and Northern Cape - Grade 6 scores increase in the first period of program delivery and then decline sharply to below significance by end-project. The exception, once again, is KwaZulu-Natal. Once again, we conclude that the impact of the IEP has been strongly concentrated at Grade 3 level and that this impact is greatest in KwaZulu-Natal. Secondly, only in KwaZulu-Natal were scores also increasing at Grade 6 level between Mid and Post 1. Consequently, the impact of the IEP is

18 Review of the Impact of the IEP 15 strongest and most significant in KwaZulu-Natal and the scores from this province dominate the mean overall project impact figures Impact at School Level Another way to look at distribution of impact is to consider the proportions of measures obtained from the tested schools - one measure is one test (e.g. literacy at Grade 3) that provide evidence of different levels of impact. Ideally, of course, one would want the clear majority of measures to exhibit positive gains in learner performance to show that intervention effect is widely and evenly spread amongst the schools, grades and subjects. However, recent research by Nick Taylor of JET suggests that learner impact depends to a large degree on the pre-existing capacity and willingness of schools or teachers - to participate in and benefit from an intervention program. Many studies show that, even when genuinely significant impacts on learner performance are measured across whole samples, this impact is typically concentrated in only some of the schools and/or for some of the teachers while others, very similar in most respects (facilities, staffing, socio-economic status), do not even though they are exposed to the same program. Figure 13: Distribution of Impact: Cohort Positive Impact No Impact Negative Impact 26.8 Looked at this way, it is evident that under half of all the IEP test measures in Cohort 1-3 yielded evidence of significant positive impact on learner performance between baseline and Post 1 while over one quarter of them indicated significant negative impact. To find out if there were any areas with which positive impact was more strongly associated than others, the table below summarizes the distribution of the five highest and five lowest gains for schools by province and by magnitude. Table 8: Distribution and magnitude of Highest and Lowest Mean Gain Scores in Cohort 1-3 Schools: Cohort 1-3: Baseline to Post 1 Highest 5 gain scores Lowest 5 gain scores Mean gain Provincial distribution Mean decline Provincial Distribution Literacy +35.4% All in KZN -18.2% 3 = EC, 2 = NC Numeracy +24.4% All in KZN -9.2% 1 = KZN, 4 = EC Mathematics +12.2% 1 = KZN, 4 = EC -10.8% 3 = KZN, 2 = NC Science +18.4% 1 = KZN, 3 = EC, 1 = NC -25.4% 2 = EC, 2 = NC Grade % All in KZN -8.8% 4 = EC, 1 = NC Grade % 2 = KZN, 2 = EC, 1 = NC -15.4% 1 = KZN 2 = EC, & 2 = NC It is evident that 12 of the 20 highest gains were in KwaZulu-Natal, considering both grade levels, while at Grade 3 level all 10 of the highest gains were in this province.

19 Review of the Impact of the IEP 16 We have by now established that the most significant and consistent conclusions we can draw about the impacts achieved by the IEP in Cohort 1-3 are that: Impact is concentrated at Grade 3 level Impact is concentrated in KZN Only in KZN were scores for both grade levels all increasing by Post 1 The scores from KZN dominate the overall project impact figures All of this evidence suggests that the search for the causes of learner performance should be sought in this province 6. The Causes of Impact on Learner Performance It is a truism that a very significant factor in understanding impacts upon learners achieved by interventions is the qualitative relationship between the strategic approach of a program and its actual delivery. Figure 14: The Generic Relationship of a Strategic Approach & its delivery DELIVERY STRATEGY Good Bad Good predicted impact likely no impact likely Bad Low/no impact likely no impact likely Consequently, uneven impact across a sample with provincial, or other, components suggests potential variation in the delivery of the intervention programme in these different components. Where impact is strongly concentrated in one component of the sample, it is very likely that causal explanations can be found in the nature of the delivery of the programme in that component. In terms of factors common to all three provinces, all of them had the same delivery agencies for numeracy and mathematics (COUNT) and science (PROTEC). Two, the Eastern Cape and KwaZulu-Natal, also had the same delivery agency for literacy (READ) while MOLTENO was the delivery agency for the Northern Cape. These programmes are all structured through the IEP strategic model and, consequently, schools have all received substantially the same, or very similar interventions. Each of the provinces was coordinated by a different management agency; in KwaZulu-Natal, MSTP for and LCD for , READ in the Eastern Cape and LCD in the Northern Cape. It is therefore possible that differences in learner impact may be related to variations in the delivery of the intervention through these agencies, though none of the current data clearly supports this possibility. The remaining factor is IEP project management and delivery in each province and here there is clear evidence of variation in KwaZulu-Natal. There is also clear evidence of impact in this province, especially at Grade 3 level. Therefore, there is a strong argument that variation in intervention delivery in KwaZulu-Natal resulted in significant improvement in learner performance.

20 Review of the Impact of the IEP Variation in Intervention Program Delivery in KwaZulu-Natal The DDSP study concluded that specific changes to intervention design and delivery resulted in significant increases in learner scores. These changes were: Supply to schools of a specific curriculum sequence (i.e. a syllabus ). Supply to schools of learner workbooks based on this sequence. Supply to schools of assessment resource banks. Increased level of school and classroom monitoring of the application of the programme. Some evidence that involvement of local-level DoE officials in monitoring schools supported impact. In the IEP, the initial KwaZulu-Natal project manager also participated in the DDSP and had a detailed understanding of the findings of the DDSP study, as well as those of the external JET evaluation. He was employed in the IEP until February 2007, supported by the current manager until from late 2006 until his departure. The internal delivery system set up in this province was, therefore, strongly based on the experience and outcomes of the DDSP. Primary components of this provincial system included: In 2006 schools were supplied with learner workbooks developed by the service providers comprising, effectively, a syllabus of study for teachers, as well as multiple and extended opportunities for learners to practice newly learned content. The Assessment Resource Banks developed for the DDSP were available in all provinces but, in KwaZulu-Natal in 2006 they were accompanied by common assessment tests developed within the IEP in the province. These common assessments were supported by the DoE, including the production of league tables of learner performance. Detailed monitoring instruments, cross-related to the IQMS performance standards, were developed and used for all IEP result areas. Starting in 2006, these instruments were used by trainers during school-based support in KwaZulu-Natal, extending to the other IEP provinces in It appears to me that this variation explains the higher degree of impact achieved by the IEP in KwaZulu-Natal from Mid to Post 1 testing. Secondly, the fact that the IEP learner workbooks and monitoring tools, and in some cases ARBs and supplementary reading materials, were supplied to schools in both of the other cohorts in 2007 largely explains the impact on these cohorts as well. Consequently, it appears safe to conclude that the evidence from both the DDSP and the IEP supports the argument that these three elements of program delivery were the most likely causes of the improvements in learner performance in both of the interventions. 7. Why did Impact Occur in Grade 3 only in Cohort 1-3 and Cohort 4a? There is data from another study that may help to illuminate this issue. In 2006, a case study for the Khanyisa School Transformation Programme in Limpopo was carried out. 4 In this study the six highest and six lowest schools mean scores for two Grades (3 and 6) and two subjects (literacy and numeracy) were selected from a population of 100 schools. Once the two research groups were identified on the basis of overall rank order, their relative performance at Foundation and Intermediate Phases for both subject areas was investigated. 4 Schollar, E. A Case study of quality in Khanyisa schools Mimeo.

21 Review of the Impact of the IEP 18 Table 9: High and Low Mean scores: Khanyisa (%) Grade 3 Grade 6 Combined Literacy Numeracy Combined Language Mathematics Combined mean Top Bottom Difference Almost all of the overall difference between the top and bottom 6 schools was caused by the mean scores at Grade 3 level. The difference at this level is +21.5% but the scores converge after that and the difference is only +3.1% by Grade 6, a decline of -18.4% in the difference between the groups at the higher grade level; the same pattern exists as in the IEP data 5 significant impact at Grade 3 level but no/very little impact at Grade 6 level. The analysis of the Khanyisa case study data suggested that some generic factor/s, operating in all of the schools, depresses the performance of learners in Intermediate Phase irrespective of how they perform in Foundation Phase. The most important of these factors appeared to be: Curriculum coverage is routinely incomplete at all grade levels and the gap between the received and intended curriculum progressively widens from one grade to the next. Assessment is ineffective in producing a reasonable match between what learners are supposed to know, and what they actually do know, at each grade level. 6 Teachers, consequently, are faced with enormous ranges of learner ability in each class and this makes it next to impossible to consistently teach to, and assess at, the Assessment Standards, for the relevant grade. The problem gets worse the higher the grade level. The persistence of inappropriate simplistic counting methods in performing arithmetical calculations extends well beyond Foundation Phase and progressively inhibits the capacity of learners to deal with more complex and cognitively demanding mathematical tasks. To these factors I would now add two other, closely linked, factors a misunderstanding of the importance of both short and long-term memory to cognitive development, combined with the associated discouragement of direct instruction in favour of methods stressing the provision of minimal instruction to learners (e.g. discovery learning and teacher-as-facilitator ) that is explicit in a constructivist curriculum. Some current international neuro-cognitive research, for example, suggests that the perceived prohibition of memorization in any form, and its conflation with rotelearning, has had severely negative effects on learner development in primary schools, especially in mathematics 7. All of these factors make it very much more difficult to obtain measurement of significant impact on learner performance the higher the grade level when standardized tests are administered the cognitive gap, as well as the associated gap between the intended and received curriculum, has widened to the point where it is very difficult indeed to achieve improvements in learner 5 The exception was the data from Cohort 4b but these scores were obtained over a very short period and are much less reliable than those obtained from the other two cohorts. 6 It will be noted that data from the National Systemic Evaluation very strongly supports this conclusion; around 80% of all learners in our schools at Grade 6 level have not achieved the minimum expected standards for this grade yet all were passed as having done so by internal school-based assessment. This enormous gap between systemic assessment and school-based assessment is a real cause for serious concern at a national level and ultimately manifests itself in the dismal performance of South African learners in the first standardized examination they write - at matriculation. 7 See, for example, Abadzi, H. (2006) Efficient Learning for the Poor: Insights from the Frontier of Cognitive Neuroscience. World Bank. And Kirschner, P., Sweller, J. and Clark, R. (2006) Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41 (2),

22 Review of the Impact of the IEP 19 performance by short-term intervention programs. Conversely, this systemic problem is, necessarily, progressively less severe the lower the grade level and this explains why the impact of interventions is more likely to occur at these lower levels. 8. Some Key Issues in Intervention Delivery that Affect Learner Performance A number of these issues will also be mentioned in the review of the different result areas of the IEP that follows but they are mentioned here as well because I believe that all of them have had a significant affect, one way or the other, on learner performance Involvement of local-level DoE officials This factor has received attention from virtually every education development program in recent years. The logic is simple and inescapable to be sustained and spread, intervention effects have to be lodged within the routine education system; as long as program interventions are localized, and distant from DoE officials, the effects they will have are, at best, of a limited life span and geographical spread. As we have already seen, there is no convincing evidence that improvements in learner performance were sustained after end-project in the IEP (or the DDSP) - with the exception of numeracy in Cohort 1-3, the only one in which it could be measured. However, this is not unique to the IEP or the DDSP by any means. If the root cause is indeed an insufficient level of engagement with and lodging in the routine system - surely a key element of the systemic strengthening sought by the IEP - it is clear that no other recent education development program has achieved engagement of local-level DoE officials to any degree of consistency either. The main reasons articulated by departmental officials are that they have many tasks to perform, and many schools with which to deal, and are often without sufficient transport facilities. They, consequently, find it difficult to give a great deal more attention and time to some schools, that are receiving external support, than to others - which are not receiving any support. In some cases, local-area offices are also short staffed, exacerbating the problem. On the other hand, it is evident in most recent programs that, while the majority of officials do not really engage with intervention programs beyond formal attendance at some events, and general approval of the project, there are typically a few officials who do so despite dealing with similar circumstances. There is also, in other words, a question of prior willingness to consistently engage with intervention programs on the part of DoE officials 8. In the IEP, interviews, and some of the field research, suggest that there have been a number of positive outcomes in this area. While it clearly has not yet crossed the threshold of likely impact on learner performance across whole cohorts, it is potentially an important change that the IEP was learning to achieve. All of the provinces have succeeded in involving district or circuit officials in training. This was not consistently achieved across whole cohorts for all officials or in terms of officials always remaining throughout entire courses. Nonetheless, the fact that the IEP succeeded to the degree that it did in attracting officials to training courses is certainly a positive achievement. In addition, there is indicative, if anecdotal, evidence of an increasing level of participation in school monitoring by local-level officials in some districts/schools. While there is no suggestion that monitoring has been uniformly achieved across all of the IEP cohorts/districts/schools, it does appear as if this is relatively more likely to happen because of involvement with the IEP. There is 8 In fact, it is equally clear that the rider applies at school and classroom level as well; many programs of which I am aware have had to deal with this factor to one degree or another.