Report Namibia Textbook Procurement Baseline Study: Report May 2010 Textbook Procurement Baseline Study
Report Namibia Textbook Procurement Baseline Study: Report May 2010 Commissioned by the Millennium Challenge Account Namibia With funding from the Millennium Challenge Corporation GOPA Consultants Hindenburgring 18 61348 Bad Homburg Germany Phone +49 6172 930-351 Fax: +49 6172 930-440 Email: hans.bonarius@gopa.de
CONTENTS Contents 1 Background... 5 2 Executive Summary...6 3 Data Collection and Processing... 7 3.1 Textbook Control Forms...7 3.2 Data entry of Textbook Control Forms...7 3.3 Data collection and processing of the...9 4 Data Analysis... 11 4.1 Comparison of Sample Survey textbook counts to Textbook Control Form.03 (inventory)... 11 4.2 Comparison of Sample Survey textbook counts to Textbook Control Form.01 (order)...15 4.3 School level analysis... 18 5 Conclusions...20 6 Recommendations...21 i
APPENDICES Appendices Appendix 1 Appendix 2 Appendix 3 Tables of potential data problems in the captured Textbook Control Forms Sample Selection Survey Instruments ii
TABLES Tables Table 1: Distribution of differences in counts (all linked records)...12 Table 2: Distribution of differences in counts (several unlikely or incomplete records removed)...14 Table 3: Distribution of differences in counts between total stock and stock counted during the Survey... 16 Table 4: Distribution of differences in counts between stock on hand and the stock counted during the Survey...17 iii
ABBREVIATIONS Abbreviations GOPA MCA-N MCC MoE NIED SIAPAC VAT Gesellschaft für Organisation, Planung und Ausbildung Millennium Challenge Account Namibia Millennium Challenge Corporation Ministry of Education National Institute for Educational Development Social Impact Assessment and Policy Analysis Corporation Value added tax iv
CHAPTER 1 Background 1 Background This report was compiled in fulfilment of the requirements of paragraph 4.4 (5) of the Terms of Reference for the Textbook Procurement Baseline Study as well as paragraph 2.4 (5) of the Contract for Consultant Services For the provision of a Textbook Baseline Study (Contract No: MCAN/CIF/RFPIP2.1): Present a report ( Report ) on findings (must include statistical calculation on accuracy of data on stock on hand data reported in the Textbook Control Form.01 - i.e. calculate an estimated margin of error on the Textbook Control Form reported data based on the Survey results). The immediate objective of the Textbook Baseline Study was to ascertain the exact level of textbooks present in each Namibian school for the subjects English, Mathematics and the natural sciences in grades 5 through 12. The Survey outcome was to assist to ascertain the accuracy of the textbook counting, recording and reporting process. In the first phase of the Survey, textbook counts were conducted in 150 randomly selected schools the 10% Sample Survey to assess the accuracy of the completed Textbook Control forms.01 and.03 submitted by schools in 2009. This report, the Report, presents findings on the accuracy of the stock on hand data reported in Textbook Control Forms.01 and.03. Tabulated order data were already provided to the MCA prior to this report. The Textbook Baseline Study Report will include additional details and cover other aspects of the Survey. 5
CHAPTER 2 Executive Summary 2 Executive Summary The was conducted in 150 schools randomly selected from all schools receiving textbooks from government. Of these 150 schools, 89 schools had submitted Textbook Control Forms.01, and 125 had submitted Textbook Control Forms.03. Data submitted by schools using the Textbook Control Forms.01 (orders) and.03 (inventory) were entered on computer using a data entry system developed by the MoE. This system was preferred by the MCA and MoE above the Consultants system, as it was intended to use the MoE system in future years. Several challenges were experienced during data entry, which had an impact on the data quality. The most important challenges were: there was inadequate data verification possible at the time of data entry; the list of books in the database, from which books entered had to be selected, contained many multiple entries, which caused the same book title to be entered against different serial numbers in the database, which in turn had a negative impact on linking different data sets. It was found that 45% of the Survey textbook counts agreed with Textbook Control Form.01 within a range of ± 2 books (78% within ± 12 books.) The corresponding figures for Textbook Control Form.03 were 45% and 74%. The standard deviations were 22.5 and 35.8 respectively for Forms.01 and.03. A relatively high number of apparent outliers were observed (see from page 11 onwards) and will have contributed to the high standard deviations. Some of these outliers might have been caused by the challenges experienced in capturing the data. As an example of data problems observed: Among the 1 136 book counts in the Survey which could be linked to an entry in Textbook Control Form.01, 156 entries had not reported the stock on hand. It was concluded from the analyses that the data did not suggest any deliberate large-scale incorrect reporting by schools. Whether or not the observed accuracy of the data reported by schools met the needs of the MoE and MCA depended on the needs of the users of the data. The accuracy was expected to be adequate for the general use of the data, but inadequate in critical cases, such as placing orders very closely meeting the needs, or establishing a baseline for a good book stock record system, if it was intended to introduce such a system. It was recommended to consider the required accuracy and completeness of the data in deciding on conducting a full (100%) textbook Survey. It was also suggested that good training of key staff in completing the Textbook Control Forms may be sufficient for attaining adequately accurate reporting by schools. Improvements to the Textbook Control Forms and the data entry system were recommended. 6
CHAPTER 3 Data Collection and Processing 3 Data Collection and Processing A brief description of the data collection and processing, as well as challenges experienced in this regard, are reported in this section. It is important to take note of the challenges, as they have a significant impact on the data analysis and interpretation. 3.1 Textbook Control Forms The MoE had developed two Textbook Control Forms to be completed by schools: Textbook Control Form.01 was used for ordering textbooks. One form had to be completed per combination of school, grade and subject. Apart from the book title, ISBN and ordering information, the form provided columns for reporting for each ordered book the stock on hand and the stock to be written off. Textbook Control Form.03 was used for reporting the inventory of textbooks in each school, again completing a form for each combination of school, grade and subject. The Textbook Control Forms.01 and.03 were sent to schools by the MoE, requesting the schools to complete these forms. The return rate was low, which prompted the MoE to send forms for a second time. Although the schools were requested not to submit their forms again if they had already done so during the first round, a considerable number of schools completed and submitted a second set of forms. Many forms were submitted late or incomplete. By the time the data analysis started, 125 sample schools had submitted Textbook Control Forms.03 (inventory), of which four schools had not submitted any forms in the Survey subjects, English, Mathematics and the natural sciences. It was finally determined that only 117 schools had reported books which were also found in the Sample Survey. Textbook Control Forms.01 (order) had been submitted by 89 sample schools, one of which did not submit any form in the Survey subjects. When the Survey was conducted, Textbook Control Forms were still received from schools. Excluding schools from the sampling frame, which had not submitted their forms by the start of the Survey, would have introduced a bias to the Survey. 3.2 Data entry of Textbook Control Forms The Textbook Control Forms were to be completed by all government schools, and schools receiving textbooks from the government for all grades and subjects. The MoE thus extended its data collection beyond the subjects and grades included in the Survey. The data enterers hired by the Consultants and data enterers hired by the MoE shared a training facility of the MoE for the data entry. Textbook Control Forms were still being received from schools for weeks after data entry had commenced. 7
CHAPTER 3 Data Collection and Processing The conditions of data entry, in particular the sharing of a facility with another group working on the same sets of documents, the constant delivery of additional forms, and the difficulties experienced with the data entry system, were relatively challenging. The time constraints for completing the data entry, combined with the tedious process of entering data on the computer system, aggravated the situation. It was not possible under these conditions to keep proper records of the deficiencies observed in the completed Textbook Control Forms. The data entry system for capturing the Textbook Control Forms was developed by the MoE. The Consultants had developed a different system, but the MCA and MoE preferred using the MoE system, mainly for sustainability reasons, i.e. to ensure the continued utilisation of the system by the MoE after the end of the Survey. Several features and shortcomings of the data entry system have to be reported here, as they had a significant impact on the data: The system provided inadequate data verification at the time of data entry: It was possible to enter non-existent school codes, i.e. it was possible to enter data using a school code with no school associated with that code. Those data could only be deleted. Attempts were made later to identify missing data and to capture those forms again for which there were no data in the system. It was possible to enter the same data more than once. Duplicate data were identified later, and redundant records were deleted. There was no check on the correctness of the data. If provision had been made on the textbook control forms and in the system to enter horizontal and vertical totals, then these totals could have been utilised for verifying the consistency of the data. Incorrect data entry was observed in a number of cases when the entered data were compared to the original forms while deleting duplicate entries. Books were entered on the system by selecting the appropriate book from a long list of 6 245 books. This list was compiled from the NIED book catalogues for the years 2005 to 2009. Although the data entry system automatically moved to possible entries as the ISBN or book title was entered, the selection of a book entry was still very tedious, time-consuming and error-prone. The following problems were experienced: Certain books could not be located at all, that is, neither the title nor the ISBN could be located. This was to some extent due to having to enter the ISBN or title starting with the first number or character. The method often failed as a result of the switch from 10 to 13-digit ISBN, or because schools had not entered the title on the form in exactly the same way as it was stored in the system. A better facility for looking up books was provided at a later stage, which resolved some of the shortcomings. When a book could not be located in the list of books, it could not be entered at all, resulting in a considerable number of reported books not having been entered. There were cases where books legitimately could not be found on the system, as they had not been included in the NIED textbook catalogues since 2005. The list of schools included many multiple entries. These duplicates originated from the list having been compiled from different years textbook catalogues. In some cases, books also appeared in a textbook catalogue under different headings. The duplicate entries resulted in a serious problem during the data analysis: The book to which each data record referred was identified by the serial number of the book in the list of books. In cases where there were 8
CHAPTER 3 Data Collection and Processing several entries of the same title or ISBN, each of these had a different serial number, i.e. they appeared to the computer system, and thus also in the analysis, as different books. This was particularly difficult when comparing different data sets, as they could no longer be linked reliably. A conversion table was later compiled by the Consultants with the assistance of NIED to resolve this problem to some extent. In a number of cases, however, it was not possible to determine whether or not certain records referred to the same book. In some of these cases, the ISBN had not been state;, in others, the same ISBN was apparently used for a learner textbook and the corresponding teacher s guide. The conversion table therefore did not fully resolve the problems caused by duplicate entries in the list of books. After data entry of the Textbook Control Forms was completed, several lists of potential errors or omissions were compiled 1 and verified, comparing the original forms submitted by schools to the data entered in the database. In this way it was ensured as far as reasonably possible that all submitted forms had been captured, and that they were not duplicated. Duplicate book entries were again attended to during the data analysis, taking into account the above-mentioned conversion list. Corrections were made, again comparing the entered data to the original forms. The measures described above could not resolve all possible data issues. Undetected incorrect data could, for example, have been caused by a data enterer erroneously having made an incorrect choice of book title due to a missing or incorrectly recorded ISBN on the form, or because a book title was abbreviated on the form. As indicated above, the Textbook Control Forms and the data entry system did not make provision for data verification at the time of data entry, except for visual inspection of the data, which was carried out. 3.3 Data collection and processing of the The Survey was conducted according to the Survey Design Report, which had been submitted to and was approved by the MCA. A detailed report on the Survey is included in the Textbook Baseline Study Report. The sample selection, at the request of the MCA done in the presence of the MCC Resident Country Director, is described in Appendix 2. Two necessary replacements of schools in the sample are reported in the same appendix. Due to the small sample of 150 schools, it was agreed with the MCA during the Survey design that the sample would not be stratified. The sample design ensured that schools in all regions, of all phases and of all sizes were represented in the sample. Textbook data and enrolment numbers by age were collected in the Survey schools, using the forms attached as Appendix 3. Data were collected in the school on the class-group level, i.e. the enumerators went from class to class to count the textbooks and learners. Textbook numbers were not summarised to avoid transcription errors. The enrolment data were transcribed to a summary form and aggregated to the grade level. The design of the enrolment data collection and summary forms facilitated correct transcription, using a suitable layout and including sub-totals and totals to verify the transcription. The data were entered on a system developed by the Consultants. This system provided for considerable data verification at the time of data entry. The data collection forms had been designed, taking into account data verification at the time of data entry. 1 See Appendix 1 9
CHAPTER 3 Data Collection and Processing The data-entry system for the Survey data had to be based on the list of books used in the MoE data-entry system, as the data sets had to be linked using the serial numbers of books in the list. Some of the problems described above therefore also affected the entry of the Survey data. The facility for selecting textbooks from the list of books provided a clearer view of books. It also allowed searching for titles by any part of the ISBN or title. 10
CHAPTER 4 Data Analysis 4 Data Analysis The main purpose of this report is to determine the extent to which the numbers of books reported by schools, in particular the stock on hand reported on Textbook Control Form.01 (the order form), correspond with the number of books counted in the sampling Survey. The quantity analysed was thus the difference between the stock reported by the schools and the number of textbooks counted, determined in each school per book title. The desired difference was zero. As the sample was not stratified, no analyses were made according to different categories, such as regions or school phases, etc. The textbook inventory data collected in the Survey were compared to the data reported on the Textbook Control Forms.01 and.03. The data sets were linked for the comparison using the combination of school code, grade, subject and book serial number. That is, the numbers of copies of a specific book title counted for a particular subject and grade in a school were compared to the corresponding count in the other data set. In the initial analysis, the conditions of the books (good, fair, poor or write-off) were ignored, as the classification was partially based on the judgement of the person completing the form. Linking the data sets was affected by the multiple entries in the list of books in the database, as described above. It was therefore possible that corresponding data were not linked to one another as the data referred to different serial numbers in the list. 4.1 Comparison of Sample Survey textbook counts to Textbook Control Form.03 (inventory) The following comparison is based on the total numbers of reported books, i.e. the sum of good, fair, poor and write-off books. The difference in books reported below is the number of books reported on Textbook Control Form.03 minus the number counted in the. A positive difference indicates that the school reported more books than counted in the Survey, while a negative difference indicates that more books were counted in the Survey. Stated differently, positive differences indicate over-reporting by schools, negative differences indicate underreporting by schools. Analysis of all records which could be linked The frequency of differences in counts, calculated as described above, was distributed as follows: 11
CHAPTER 4 Data Analysis Table 1: Distribution of differences in counts (all linked records) Range Range In terms of mid-point Frequency -33 and less -33 and less 164-32 to -28-30 ± 2 34-27 to -23-25 ± 2 40-22 to -18-20 ± 2 73-17 to -13-15 ± 2 90-12 to -8-10 ± 2 175-7 to -3-5 ± 2 302-2 to 2 0 ± 2 1553 3 to '7 5 ± 2 336 8 to 12 10 ± 2 173 13 to 17 15 ± 2 109 18 to 22 20 ± 2 69 23 to 27 25 ± 2 53 28 to 32 30 ± 2 34 33 and more 33 and more 209 Frequency of differences between Textbook Control Form 03 and Sampling Survey Number of cases 1800 1600 1400 1200 1000 800 600 400 200 0-33 and less -32 to -28-27 to -23-22 to -18-17 to -13-12 to -8-7 to -3-2 to 2 3 to '7 8 to 12 13 to 17 18 to 22 23 to 27 28 to 32 33 and more Difference 12
CHAPTER 4 Data Analysis The differences between the counts were clearly centred around zero: 45% of all entries had a difference in the range -2 to 2 books, while 74% of the differences ranged between -12 and 12 books. The average difference was 0.96, that is, an over-reporting by schools of one book. The standard deviation was 35.8. As shown below, the picture changes slightly when an adjustment is made to the data by excluding entries from the analysis, which appear to be incorrect. While the differences in the counts were not very large in general, there were a number of considerable differences between the counts, well exceeding 30 books in a number of cases. Differences in counts compared to normal distribution (All entries which could be linked) 3000 2500 2000 1500 Normal distribution Histogram 1000 500 0-139.5-119.5-99.5-79.5-59.5-39.5-19.5 0.5 20.5 40.5 60.5 80.5 100.5 120.5 140.5 The distribution of the differences is not a normal distribution, as indicated in the graph above. This was to be expected, as all data or at least most were expected to be accurate. The large differences which were also observed, even though in a small number of cases, contributed to a relatively large standard deviation. The normal distribution shown in the above diagram has been calculated using this standard deviation. The above chart, showing the discrepancy between a normal distribution and the distribution of differences in textbook counts, demonstrates that a statistical calculation assuming a normal distribution will lead to incorrect conclusions: The number of observed cases with almost zero difference in counts is several times higher than the normal distribution would suggest, while, with a normal distribution, larger numbers of cases with high differences in counts would be expected. Thus, a statistical calculation assuming a normal distribution would exaggerate the differences in the counts. There may be different causes for the differences between the data sets, such as incorrect reporting, links between the data sets which failed due to the reasons described above, and data entry errors. The latter two reasons are likely to have caused the outliers (extreme differences in counts) which resulted in the relatively high standard deviation. 13
CHAPTER 4 Data Analysis It was reported above that the Survey data were collected and entered into the database by class, e.g. Grade 8A and 8B separately. This might have resulted in the same book title having been entered against different serial numbers for the two classes, without the data enterer noticing it. Differences in the numbers of books roughly equalling a typical class size were observed in several cases. There were also cases where one data set had a number representing one or several classes, while the other data set had a single-digit number, typical for teacher materials. Such differences may again have been caused by the ambiguities in selecting a book title during data entry, as reported above. An adjustment of the data was made to determine whether excluding such suspect cases would make a significant difference to the outcome: The following data were omitted: All entries where the apparent Survey count was less than 60% of the count according to Textbook Control Form.03 (this would exclude cases where the data for a significant portion of classes was not linked to the other classes). All entries where the Textbook Control Form.03 count was less than 6, while the Survey count was greater than 15 (this would eliminate records showing only teacher materials in the Textbook Control Form, while the corresponding Survey data included textbooks for learners.) The inverse of this is covered by the exclusion described under the previous bullet. The following distribution was obtained after removing such unlikely entries: Table 2: Distribution of differences in counts (several unlikely or incomplete records removed) Range Range In terms of mid-point Frequency -33 and less -33 and less 133-32 to -28-30 ± 2 29-27 to -23-25 ± 2 33-22 to -18-20 ± 2 60-17 to -13-15 ± 2 75-12 to -8-10 ± 2 169-7 to -3-5 ± 2 298-2 to 2 0 ± 2 1477 3 to '7 5 ± 2 258 8 to 12 10 ± 2 94 13 to 17 15 ± 2 46 18 to 22 20 ± 2 21 23 to 27 25 ± 2 12 28 to 32 30 ± 2 6 33 and more 33 and more 23 14
CHAPTER 4 Data Analysis 1600 Frequency of differences between Textbook Control Forms and Sampling Survey (several unlikely or incomplete records removed) 1400 1200 1000 800 600 400 200 0-33 and less -32 to -28-27 to -23-22 to -18-17 to -13-12 to -8-7 to -3 Frequency -2 to 2 3 to '7 8 to 12 13 to 17 18 to 22 23 to 27 28 to 32 33 and more Range Removing the records as described above largely eliminated the positive outliers, i.e. records showing a significant over-reporting of books by schools. This is reasonable, as it is not expected that schools report more books than they have in stock, especially when they are ordering books at the same time. Under-reporting might have several causes: for example, omitting stock in store rooms. The changes observed in the distribution after excluding some unlikely data shows that data errors may cause a significant number of the apparent outliers. In this distribution, 54% of the cases range between -2 and 2 books, up from 45%, when no records are removed. Eighty-four per cent of the cases range between -12 and 12 books, compared to the 74% when all books are included. The average is -4.8, caused by eliminating assumedly wrongly indicated over-reporting. The standard deviation is 27.0. 4.2 Comparison of Sample Survey textbook counts to Textbook Control Form.01 (order) The Sample Survey could only be linked to counts in Textbook Control Form.01 for books for which orders had been placed. There were therefore substantially fewer cases to compare than for Textbook Control Form.03. Due to the ambiguity of classifying the condition of books, only the total book counts were compared. Similar problems were experienced in linking records, as described for Textbook Control Form.03. In addition, there were cases where a revised version of a book was ordered, while the stock on hand actually referred to an earlier edition of the book. The serial number of the revised book on the order form therefore did not correspond to the serial number in the Survey book count. Out of the 1 136 cases where the Survey inventory could be linked to a book order, 156 had no book stock reported on the order form. These cases were omitted in the analysis, which is based on 980 cases. 15
CHAPTER 4 Data Analysis Table 3: Distribution of differences in counts between total stock and stock counted during the Survey Range Range In terms of mid-point Frequency -33 and less -33 and less 41-32 to -28-30 ± 2 10-27 to -23-25 ± 2 14-22 to -18-20 ± 2 27-17 to -13-15 ± 2 25-12 to -8-10 ± 2 79-7 to -3-5 ± 2 99-2 to 2 0 ± 2 437 3 to '7 5 ± 2 99 8 to 12 10 ± 2 53 13 to 17 15 ± 2 24 18 to 22 20 ± 2 14 23 to 27 25 ± 2 9 28 to 32 30 ± 2 12 33 and more 33 and more 37 The general pattern of differences between the numbers of books was similar to the pattern found for the inventory. Similar to the analysis of the comparison of the textbook inventory without excluding doubtful data, 44.6% of all cases ranged between a difference of -2 and 2 books. There were 767 cases in the range -12 to +12 books, i.e. 78.3% of the cases. The average difference was -1.1, which means a slight over-reporting of books. This apparent over-reporting seemed to have been caused by the relatively high number of negative outliers. The standard deviation was 22.5. Frequency of differences between stock reported on the Order Form and in the Sampling Survey Number of cases 500 450 400 350 300 250 200 150 100 50 0-33 and less -32 to -28-27 to -23-22 to -18-17 to -13-12 to -8-7 to -3-2 to 2 3 to '7 8 to 12 13 to 17 18 to 22 23 to 27 28 to 32 33 and more Difference between book counts 16
CHAPTER 4 Data Analysis The reported stock on hand is compared below to the Survey counts. It was possible that the Survey teams would not be able to see all books in the one or two days they visited each school. Normally, this would have happened when pupils left their books at home, and where the homes were too far away from the school to send the pupils home to fetch the books. In such cases, the condition of the books could not be determined. These books (602 out of a total of 22 532 included in this analysis) were counted as good or fair. Table 4: Distribution of differences in counts between stock on hand and the stock counted during the Survey Range Range In terms of mid-point Frequency -33 and less -33 and less 36-32 to -28-30 ± 2 10-27 to -23-25 ± 2 8-22 to -18-20 ± 2 18-17 to -13-15 ± 2 23-12 to -8-10 ± 2 64-7 to -3-5 ± 2 85-2 to 2 0 ± 2 438 3 to '7 5 ± 2 97 8 to 12 10 ± 2 75 13 to 17 15 ± 2 37 18 to 22 20 ± 2 29 23 to 27 25 ± 2 10 28 to 32 30 ± 2 12 33 and more 33 and more 38 In 438 out of the 980 cases (combinations of school, grade, subject and textbook title), i.e. 44.7%, the number of books counted in the Survey agreed within ±2 books to the number reported on Textbook Control Form.01. In 77.4% of the cases, the numbers agreed within ±12 books. These percentages represent the estimated percentages within which the stock on hand was reported correctly within the stated limits. The average difference was an insignificant 0.5 books over-reporting, while the standard deviation was 22.6. 17
CHAPTER 4 Data Analysis 4.3 School level analysis It can well be conceived that certain schools would be very diligent in reporting textbook numbers, while others might be to some extent careless, or even deliberately provide false data. Motives for deliberate false reporting could be hiding stock losses, or attempts to obtain more books. If a school were wanting to hide book losses, it would report more books than it has; and if it attempted to obtain more books than needed, it would report fewer than the actual number of books. An analysis was made of the average differences in book counts in each school, comparing the Survey count to the Textbook Control Form.03 count, to determine whether the data suggest such behaviour. The average differences in book counts were calculated for each school by calculating for each book title the difference between the number reported on Textbook Control Form.03 and the number counted in the Sample Survey. In the first chart, the normal average was calculated where a positive difference for one title would balance to some extent a negative difference for another title. Schools which exaggerated their counts for most titles would show a positive average difference, and vice versa for schools which under-reported on most titles. The second chart shows the distribution of schools according to the average absolute differences. The absolute value of the difference in counts was calculated for each book title, i.e. positive and negative differences were treated as the same. The purpose of this chart is to give some indication of the incorrect reporting without balancing out over-reporting with under-reporting. It should be noted that these charts show numbers of schools, whereas the above charts showed numbers of book counts per book title, grade, subject and school. The patterns suggested by the charts may thus be different, as small schools with only a few titles had the same weight in the charts below as large schools with many book titles. Number of schools according to their average difference in stock counts 60 50 40 30 20 10 0-27.49 - -22.5-22.49 - -17.5-17.49 - -12.5-12.49 - -7.5-7.49 - -2.5 Number of schools -2.49-2.5 2.51-7.5 7.51-12.5 12.51-17.5 17.51-22.5 22.51-27.5 27.51-32.5 Average difference betw een book counts 18
CHAPTER 4 Data Analysis Number of schools according to their average absolute difference in stock counts 35 30 25 20 15 10 5 0 0-5 5.1-10 10.1-15 15.1-20 20.1-25 25.1-30 Number of schools 30.1-35 35.1-40 40.1-45 45.1-50 50.1-55 Average absolute difference between book counts The above charts show that most (78.6%) of schools had an average difference between the book counts between -7.5 and 7.5. In 43.6% of the schools the difference was between -2.5 and 2.5. There seemed to be a tendency in several schools to reporting more books than the number of books counted in the Survey. In general, the first chart does not suggest any large-scale deliberate incorrect reporting in the one or other direction. Analysing the average absolute differences in counts, as shown in the second chart, indicates that the counts were generally not very accurate. Only 26% of the sample schools (31 out of the 117 which could be compared) had an average absolute difference in counts of less than 5. To some extent, these findings seem to be contrary to earlier findings. However, it must be taken into account that this last analysis is based on the number of schools and not on the number of book counts, like the other analyses. 19
CHAPTER 5 Conclusions 5 Conclusions In coming to conclusions, it must be kept in mind that the problems in processing and analysing the data, described above, are likely to have affected the outcomes of the comparisons. Visual inspection at the time of data entry could only partly prevent these problems, as most are related to inherent problems in the data-entry system and thus only emerged during processing and analyses. The analyses do not suggest any large-scale, deliberate, incorrect reporting by schools. The extent to which the accuracy of the data reported by schools meets the needs of the MoE and the MCA depends on the purpose the data are used for, and the level of accuracy required by the users: As a general indication of the level of provision of textbooks, the accuracy should suffice. Whether or not the accuracy meets the requirements for book orders, must be decided by those approving the orders. When orders are placed on a tight budget and an equitable distribution of books is a high priority, a higher level of accuracy would be required. If limited sharing of books in some schools, and small redundancies in other schools can be tolerated, the accuracy of reporting would be acceptable. If the MoE intends developing a good book stock record system based on the collected data, then the reported numbers would not meet the requirements of a reliable baseline. The return rate of the Textbook Control Form.03 would also be inadequate. 20
CHAPTER 6 Recommendations 6 Recommendations 1. The possible need for conducting a full (100%) Survey of schools was stated in the Terms of Reference of the Survey. It is recommended to consider the need for such a Survey, taking into account the following aspects: the required accuracy of the data, as discussed above under the heading Conclusions; the time available for collecting the data and the completeness expected the returns of the Textbook Control Forms suggest that it is unlikely to obtain a 100% return rate within a reasonably short time by only repeating the data collection done in 2009; the level of success which can be anticipated by using the alternative approach of providing good training in the completion of the forms to school principals and teachers or school administrators responsible for providing the data; the available resources for a full Survey. The above considerations suggest that a full survey of schools would have limited value for the MoE. It is recommended in the Textbook Baseline Study report to first develop a database system for the MoE to keep proper textbook stock records, and then to verify the textbook stock at all schools, to the greatest extent possible using MoE stock control officers for this exercise. 2. The data collection instruments, i.e. the Textbook Control Forms, should be revised: more space needs to be provided for entering complete book titles; column and row totals of the stated numbers should be included to provide a mechanism for verifying the data entered in the database at the time of data entry. (Provision will have to be made in the manual procedures for checking the column and row totals prior to data entry in the database). 3. The data entry system should be improved: the system should, in general, be more user-friendly: any redundant repetitive entering of data, e.g. school, grade or subject codes, should be avoided the best aspects of the MoE system and the system used for the Survey should be retained, still improving what can be improved in both systems; the list of books from which book titles are selected must be updated: no book entry should be duplicated; all books still in use in schools should be included; the database structure should be improved to preclude entering any invalid codes and to include other information on books which may be required in the utilisation of the data; as much data verification as possible at the time of data entry should be introduced, without unduly slowing down data entry. 21
Appendix 1 Tables of Potential Data Problems in the Captured Textbook Control Forms
Appendix 1: Tables of potential data problems in the captured Textbook Control Forms The following reports were generated to identify potentially incorrect, incomplete, missing or duplicate data. The printouts generated from the captured data were compared to the forms submitted by schools and any discrepancies between the captured data and the data on the original forms were corrected in the database. The scope for trapping data errors was confined to aspects which could logically be deduced from the available data. Report Identifying fields Notes Duplicate book entries (separate reports for the two Textbook Control Forms) Inventory reported in a subject, but no order placed Invalid school codes Inconsistencies between grades entered and the grades for which books were intended Excessive ratios between enrolments and books ordered (added to books on hand) Inconsistencies between the subjects for which books were ordered and the subjects against which the books appeared in the book catalogue Zero orders (row entered on Textbook Control Form 01 with ordered quantity = 0) Zero enrolments Region, School, Grade, Subject, Book Id Region, School, Grade, Subject Subject, Grade, invalid School Code Region, School, Grade, Subject, Book ISBN and Title Region, School, Grade, Subject, Book ISBN and Title Region, School, Grade, Subject (for which books were ordered), Book ISBN and Title, Catalogue Subject Heading Region, School, Grade, Subject, Book ISBN and Title Region, School, Grade, Subject The list was later refined to also intercept cases where a book title was duplicated under different Book Ids. Most cases were correct this list was intended to identify forms which had initially been captured under an incorrect school code The intended grade (or grade range) was available from the book catalogues The entries were only changed when they differed from the orders submitted by schools In most of the listed cases the subject code was entered incorrectly few book titles were incorrect 16
Appendix 2 Sample Selection
Appendix 2: Sample selection Sampling frame All government schools and private schools receiving textbooks from government. Basis of sampling All schools in the sampling frame had the same probability of being selected. Special considerations The sample was be drawn to include o schools in all regions, o primary, combined and secondary schools, o schools of all sizes. Some of the resulting strata were too small to allow a meaningful statistical analysis. Such an analysis was required and was, thus, of no concern. The combinations of regions, types of school and school sizes only served to ensure that the whole spectrum of schools was represented in the sample. Methodology The number of schools to be sampled in each region was approximately proportional to the total number of schools in the region which fall within the sampling frame. (A proportional representation was not required, as the sample was not stratified by region.) In each of the regions, the schools were ordered by type (primary, combined, secondary) and enrolment. Schools were then be selected at equal intervals, starting from a randomly selected school, determined using a random number generated by Excel. When the last school near the bottom of the list had been selected, the process continued, circling to the beginning of the list. The process described in the preceding paragraph ensured that each school had an equal probability of being selected, while a good balance of schools of different types and sizes was obtained. Replacement It became necessary in two cases to replace a school by another one. When the first replacement became necessary, it was agreed with the MCA and MCC to replace the school with the next school on above-mentioned ordered list of all schools in the sampling frame. The same method was to be applied for any other replacement. Two schools had to be replaced: School 7923, Cabatana Primary School in the Oshana Region could not be surveyed because of it is a private school not receiving textbooks from government, that is, because it did not belong in the sampling frame. The school was replaced by the next school on the school list for the Oshana Region, 8445, Ondelekelama Primary School. School 7276, Tjwatama Primary School in the Kavango Region had closed down completely (there was no teacher and the learners had moved to another school) and was, thus, replaced by school 7544, Usivi Junior Primary School. 17
Appendix 3 Survey Instruments
Appendix 3: Survey Instruments The two survey instruments on the next pages were utilized in the 10% sampling survey. 18
Textbook count form School code:... School:... Region:... Grade:... Subject:... (... ) Gd 11/12: H O Enrolment in subject:... Class:... Teacher materials (Tick when teacher materials are recorded on the form. See guidelines.) Enrolment expected 2010:... Type of book (Tick one) Number of textbooks by condition and use Stock good Stock fair Stock poor Stock write off Total stock ISBN Title 1. Textbook 2. LSM 2 3. Reader 4. Workbook 3 5. Atlas 6. Reference 4 In use Stored Total In use Stored Total In use Stored Total In use Stored Total In use Stored Books not seen Total 1 1 2 3 4 5 6 2 1 2 3 4 5 6 3 1 2 3 4 5 6 4 1 2 3 4 5 6 5 1 2 3 4 5 6 Total: Condition of book: Good: New, one year old or undamaged Fair: Usable for a further 2 years Poor: In last year of service Write off: No longer usable 2 LSM: Learning Support Materials (only include printed materials and LSMs in the textbook catalogue) 3 Workbook: Printed book into which learners write, and which should only be used once. 4 Reference: Dictionary, Encyclopaedia, etc. Note: No book may be counted more than once, e.g. readers used by different learners at different times in the year, must only be counted once!
Number of learners by age and sex Class-level form Grade:... School code:... School name:... Region:... Males Class A B C D E F G H I J K L Total Females Class A B C D E F G H I J K L Total 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Age Age 23 (+) 23 (+) Total Total
Number of learners by age and sex Summary form School code:... School name:... Region:... Males Grade 1 2 3 4 5 6 7 8 9 10 11 12 Total Females Grade 1 2 3 4 5 6 7 8 9 10 11 12 Total 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Age Age 23 (+) 23 (+) Total Total