The Impact of School Quality on Educational Performance:

Transcription

1 The Impact of School Quality on Educational Performance: A Study of Middle School Education Reforms in Beijing s Eastern City District by Lai, Fang University of California at Berkeley September, 2004 Abstract What impact does school quality have on educational outcomes, and what aspects of school quality matter most, remain hotly debated issues in the economics of education. Ambiguity mainly comes from empirical difficulties in separating school effects from the role of various confounding unobserved student characteristics. In this paper, we are able to identify the role of school quality by taking advantage of a unique natural experiment deriving from an ambitious education reform in recruiting new middle school students through a partially randomized assignment undertaken in 1998 by the Eastern City School District of Beijing. By comparing the performance of students with the same situation in the school assignment procedure but who were randomly assigned to different schools, and relating it to relevant indicators of school resources, we are able to measure the impact of school quality on educational performance without being affected by confounding unobserved individual characteristics. We address three issues. First, we test whether the education reform reached its goal of equalizing student performance across middle schools through random assignment. Second, we test whether school quality, including especially teachers qualifications and experience, explains differences in school performance. Third, we measure the relative importance of school effects versus students individual characteristics in explaining performance in national examinations. We find that, after the reform, differences in performance across middle schools have indeed been sharply reduced. We also find that school effects, especially differentials in teacher qualifications, do play a significant role in explaining students test scores. However, we find that individual characteristics play a larger role than school effects in determining students performance. 1

2 1. Introduction The impact of school quality on students performance remains a highly debated issue in the education literature. Measuring this relation, and identifying which aspects of school quality matter most, is important for developing countries with tight budget constraints if they are to use these resources efficiently for human capital formation. In China, shortage of resources in the public school system has been a concern for a long time. This shortage has multiple and complex causes and is not likely to be resolved in the near future. As a consequence, a priority task for policy makers is to allocate limited public funding to where it has the highest marginal impact on educational achievement. Aspects of school quality that may deserve priority are the construction of school infrastructure and laboratory facilities, improving methods of student tutoring, providing more on-the-job training to teachers, and improving teachers incentives and work conditions so as to make the profession more attractive to qualified candidates. In past attempts at measuring the impact of school quality on educational performance, claim has been made that the so call school quality effect may simply reflect the effects of peer-groups in the school and of children s unobservable individual characteristics. Previous research has not offered a clear answer to this question. One of the main reasons for this gap in knowledge is the lack of random assignment of peer group and school quality to enable rigorous identification of their effects. This paper explores the importance of school quality on educational achievement, benefiting from a natural experiment whereby the city of Beijing proceeded to randomly assign children to neighborhood public middle schools in 1998 as an element of school reforms. We use data from a recent census of Beijing s Eastern City District. The data include information on some 7,000 middle school students and their friends who were assigned three years ago to middle schools of diverse qualities through a partially random procedure. We utilize the random assignment of children across schools to control for the nonrandom factors in school application, and thus compare educational achievements among students who are similar but enter different schools, thus identifying the effects of school quality on performance. We also 2

3 decompose the overall school effect into different aspects of school quality and draw implications on the effect of each of these aspects on students school performance. 2. The Educational Reforms as a Natural Experiment 2.1. Middle School Education in the Eastern City District Before 1998, public middle schools (that offer the compulsory three years of middle school education) were categorized into five grades on the basis of a comprehensive measure of school quality. These grades were formally assigned by the Education Bureau of Beijing and had been kept consistent for many years. The higher grade schools, in addition to receiving superior teaching resources and infrastructure, had priority in choosing new teachers from the pool of graduates assigned to the education system. They were also in a superior position regarding the allocation of public funds and receipts of private donations. Most importantly, they could also select the better students from the pool of applicants by requiring a higher minimum test score in the district-level Uniform Comprehensive Middle School Entrance Examination. And students, understandably, preferred to attend schools with higher grades. Thus, given that tuition fees were almost uniform among all middle schools, higher grade schools were able to get the better students. As a result, disparities among middle schools in comprehensive quality measures including school reputation, school resources, and student qualities were very large. In 1998, a district-wide education reform was launched, with two objectives. One was to reduce excessive pressures on primary school children in preparing for the middle school entrance examination. This was considered detrimental to the quality of primary school teaching because it made primary school teaching examination successoriented, instead of seeking more comprehensive education based on students own interests and abilities. The second was to equalize access to educational opportunities at the middle school level by reducing disparities in the quality of middle schools. Differential school qualities depend not only on differential access to resources, but also 3

4 on differentials in student quality which in turn affects school quality through peer effects. Besides equalizing access to public resources across schools, the reforms sought to achieve these objectives by introducing two important changes in the school admission procedures. The first consisted in eliminating the middle school entrance examination. This had the objective of not only relieving pressures towards examination achievements in primary school but also of avoiding excessively early decision on students future careers which are strongly dependent on which middle school a student enters at age 12. Excellent performance in the Middle School Entrance Exam at age 12 may not demonstrate a child s potential in study. Thus, deciding students access to school resources (which to a great extent affects their future development) totally on the test scores in a single exam was considered unfair. The second consisted in using a random assignment of children across middle schools in their neighborhoods. This was considered more socially fair than sorting students by scores in the middle school entrance examination. Random assignment would also equalize quality in the pool of students across schools, thus reducing the role of peer effects on school quality. Schools, however, allowed some students to be admitted without randomization if their parents were employed in a particular school, if the students had received at least a city-level prize in academic or special skill achievements, and if direct payment of fees to the school was made. Randomization was thus incomplete, with a fraction of the students escaping the random drawing process. Fees collected by schools could also contribute to maintaining inequality in school quality in spite of equalization in the allocation of public resources. As a consequence of these reforms, the school grading system was eliminated. The district was divided into several school neighborhoods (the division in school neighborhoods was different each year) according to primary school enrollment as well as geographic proximity. Students in each neighborhood were eligible to up to seven middle schools and some middle schools were available to more than one school neighborhood. Usually, the former best schools were available to more than one school 4

5 neighborhood, while lower quality schools were only available to the school neighborhood of proximity (i.e., to primary schools near to them). In a given neighborhood, the available middle schools each have to fulfill a certain enrollment quota of children from the neighborhood corresponding to a target enrollment assigned by the Education Bureau. Eventually, the sum of target enrollments for all middle schools available to this school neighborhood should be equal to the number of students graduating from this neighborhood in the current year. A student can apply to some or all of the middle schools available to his school neighborhood, but he has to rank them from my first choice through, say, my seventh choice (according to the number of middle schools he applied to). The enrollment procedure, instead of the previous sorting-by-scores, was consequently made partially random. The procedure that was introduced is as follows. First, a 10-digit random number generated by computer is assigned to each student graduating from primary school in the current year, regardless of school neighborhood and their applications. Students are ranked by the random numbers assigned to them. The middle schools, then, first choose among the students who declare them as my first choice, enrolling students with the lowest numbers first, until they fulfill their target enrollment in the specific school neighborhood. If they cannot fulfill their target with the pool of first choice, they go to the pool of second choice (excluding those students who report the school as second choice but have already been enrolled by the schools they list as first choice ), and then the third choice and so on, until they meet the target in that school neighborhood. Students who did not get their most preferred choice are then transferred to their next preferred choice, according to the order they report during the application. If, by bad luck, a student misses all the school he applied to, he is randomly assigned to any middle school available in his school neighborhood which has not yet fulfilled its enrollment target. Thus, conditional on the school s neighborhood assignment and students applications, the enrollment procedure is a partially random assignment independent of a student s own characteristics and family background. Students of diverse academic 5

6 records and ability are thus mixed together. They are expected to spend the three years of junior middle school life together, providing a melting pot that erases differential peer effects in affecting the quality of schooling Data To asses the impact of these reforms on school quality and the remaining effect of school quality on student performance, a census was conducted in early 2002 by the Education Bureau of Beijing s Eastern City District concerning students school life and social backgrounds. The census covered all students currently enrolled in Junior 3 of middle school (i.e., students who entered middle school in 1999), their parents, their teachers, and the middle schools. It involved all 27 public middle schools affected by the education reform, and all 7,000 students in Junior 3 of these schools and their parents. There were two middle schools in the public system that were not affected by the education reforms. They were traditionally Grade A schools and, after the reform, they only kept their high school sections while giving away their middle school sections to semi-private sectors. These two schools, as well another six private or professional middle schools, were not included in the census because they were not involved in the education reform. Thus, the census covers the students (and their families) in Junior 3 whose middle schools were affected by the education reform. In the census, students were asked questions concerning their opinions about their study environment, living conditions, allocation of time to different subjects, and attitudes toward school and society. There was also a questionnaire for the students parents to fill out, asking information about the household s living conditions, wealth, parents education level, employment status, and their contributions in nurturing their children. The census data were supplemented by a survey of around 600 middle school teachers who are currently or were once in charge of the students who graduated in The teacher s survey data include (1) basic characteristics of the teachers, such as gender, age, professional rank, education, and experience, and (2) attitude 6

7 characteristics demonstrated by their responses to questions regarding school quality and their satisfaction with their current jobs. Middle school principals were also interviewed regarding school resources and changes in those during the last three years since the reform. Because, by the time the teacher s survey was conducted, three of the middle schools had been merged with other middle schools, and most of their teachers had been dismissed, the teachers survey was conducted in only 24 of the 27 schools. The census data were also supplemented with administrative data on students (1) primary school graduation test scores in two subjects, Chinese and Math, (2) test scores on all subjects all through the six semesters of junior middle school, (3) junior middle school graduation test scores, and (4) high school entrance test scores. The last two exams are formal district-uniform exams. However, all students are required to take the graduation tests in all subjects, while only those who intend to enter high school will take the high school entrance exam. The participation varies from 46% to 82% across middle schools. Moreover, for those who took the high school entrance exam, we are currently able to get only a sample of test scores for two-thirds of the students because of missing information in the application brochure which prevents us from matching test scores with students. Missing test scores due to missing information in the application brochure should be random, and thus would not affect the results. Missing test scores due to non-participation to the exams are, however, endogenous choices. Unfortunately, currently there is no information available to identify whose test scores are unavailable due to missing information, and whose test scores are unavailable due to non-participation to the examination. However, because students usually choose not to go to high school and thus not to take the exam only if they think they are unable to pass, the examination participation rate is positively correlated to the general performance of the school. Thus, ignoring the attendance issue will underestimate the school effects because the observed test scores have less variation across schools than the actual test scores. Although this may affect the precision of the results of school effects, it is not likely to invalidate the results if the findings support the existence of school effects. In spite of this difficulty, the High School Entrance Exam has the 7

8 advantage of being not only uniform across the district, but also graded by one single grading committee appointed by the Education Bureau. Other exams, which are taken by all students, though also uniform at the district-level, are graded by the schools themselves, introducing possible biases across schools. Thus, we will use the more reliable High School Entrance Exam test scores for our major analysis. We also use school administrative data that contain official records on the student s responsibilities in student societies, family structure, home address, and primary school enrollment. 3. The partial randomization process of students assignment to schools 3.1. The process The district has 27 schools that serve 7102 students students avoided the selection process put into place by the reform. These students obtained privileged access to the school of their choice, based on special conditions given to children of teachers who work in that school, children whose parents have particular links with the school, or children of parents that made financial contributions to the school. Almost all of these students that came from the district enrolled in a good school (1133 out of 1285 direct and transfer entries), indicating that these privileged access conditions were mostly used to ensure entry into a good school rather than for any other convenience. The other students went through the selection process described in this section and illustrated on Figure 1. Each student provides a list of schools, by order of preference. Students are not completely free to sign in for any school, but restricted in their choice depending on the neighborhood they live in. There are 15 neighborhoods and each one has access to 4 to 7 schools. Using a classification of schools into three levels (most popular or A schools, intermediate or B schools, and less desired or C schools) that we establish later in this section, we can verify that this geographical allocation was relatively fair, with all neighborhoods given access to at least 2 and often 3 good schools, and at least 1 of each B and C schools. 8

9 Of the 27 schools, 15 could accommodate all the students that selected them as first choice. The 220 students that chose them were thus directly assigned to their first choice. The remaining 12 schools had more first choice applicants than they could accommodate in at least one neighborhood. These most coveted schools, classified as A schools, thus proceeded to a random choice of their students among applicants. 1 This Step 1 randomization allocated 1800 students to their first choice schools, and rejected For the 2550 students that did not get their first choice, the process was repeated with their second choice. If their second choice was one of the A schools, that had filled in the first round, it was considered as invalid and they missed this round. If their choice was a school that could accommodate all the applicants, this is where they were enrolled. If their choice was for a school that received more applicants that it could accommodate, the school proceeded to randomly select its students (Step 2 randomization). All the remaining unallocated students (those with invalid second choice and those randomly selected out of their second choice) then proceed with their third choice in a similar way, and this until all students were assigned. In many neighborhoods, all the second choices other than A schools could be accommodated, and the Step 2 randomization only occurred at the third choice. 2 We label the schools that could accommodate all first choice applicants but eventually had to apply a randomized selection of students in later rounds as B school. The remaining least popular schools that never had a demand in excess of their quota are classified as C schools. It turned out that no children had to go through more than 2 randomization processes. To simplify the exposition, we have thus regrouped all the rounds of selection based on choice 2 and above on Figure 1. In summary, out of the 2550 students that entered the second round, 556 had chosen a B school that could not accommodate all their applicants, and had to go through the Step 2 randomization 1 One school that serves four neighborhoods had applications in excess of their quotas, and was thus classified as A school in two neighborhoods, while it could accommodate all applicants in the two other neighborhoods. 9

10 process. Of them, 211 were admitted in the school of their choice. The other students were assigned to a C school. Among the students that never faced a second randomization, 508 chose B schools in rounds before they filled up, and 1486 either chose a C school that accommodated them or had only invalid choices (schools that were already full) and hence ended up in C schools. This procedure thus exhibits randomization at two particular points. The Step 1 randomization concerns the students that selected for their first choice a school of category A, and the Step 2 randomization for the children that were not selected in their first choice and eventually chose a school of category B (for most students as their second or third choice) Selection channels The choices expressed by the students reveal their preferences, an important characteristic that may be associated with their performance, and should therefore be controlled for when comparing students enrolled in different school. The number of different sequences of seven choices is, however, very large, too large to be used in the analysis (particularly because of the many incomplete sequence and sequences with repetition of the same school). We consequently proceed to classify all these choices into 137 selection channels that completely characterize the selection through which the student reached the school he is enrolled in. Each selection channel is represented by three schools {s1 s2 s3}. School s1 is the student s first choice (one of the 12 A school). School s2 is the second or higher order choice, if it led to a Step 2 randomization (necessarily one of the B schools), and 0 otherwise. And school s3 is the chosen school in which the student will be enrolled if he misses the preferred schools in the randomizations he faced. Students from neighborhoods with more than one C schools that did not select any C school among their choices have s3 = 0. 2 One school of one neighborhood reached its quota only at the 5 th round, and thus operated a random selection only at that level. 10

11 We illustrate the process in neighborhood 10, in which all step 2 randomizations took place on the second choices. Neighborhood 10 has access to three category A schools, A1, A2, and A3, one B school, and one C school. Students that chose {A1 B C} thus faced a selection process that potentially entails two randomizations. As schools A1 and A2 were filled in the first round, they were considered invalid whenever selected as second or higher choice. As School B was filled in the second round, it was considered invalid whenever selected as third or higher choice. Hence, we need only show the first two choices of a student s total 7 choices to demonstrate how these different sequences are fitted in a relatively smaller number of selection channels. Here, we include students first three choices to make it more clear. For example, students that chose {A1 A2 C} or {A1 A3 C} as their first three choices were de facto facing the same selection process as those that chose {A1 C B}, {A1 C A2}, or {A1 C A3}. Both types of students were randomized on their first choice, and, if they were selected out of school A1, would automatically be enrolled in school C. Similarly a student that chose {A1 A2 B} or {A1 A3 B} would be randomized on his first choice, and, if he was selected out, would skip the second round as his second choice was not valid, not be considered for school B which, by round 3, was full, and thus ended up in school C. All seven sequences of choices thus confront the same selection process that we can summarize by {A1 0 C}, meaning students were randomized for entry in school A1, and, if they were selected out, were automatically enrolled in school C. Besides, even if students didn t choose school C explicitly in their application, for example, even if they chose {A1 0 0} or {A1 B 0}, if they missed both rounds of randomization, they will be placed in school C, thus attributed to channel {A1 0 C} and {A1 B C}, respectively. Some students selected the same school for several choices, as illustrated by choices {A1 A1 C} and {A1 B B}. Table 1 summarizes the choices that students from the 10 th neighborhood made, and the assigned selection channels. The selection process in this neighborhood of four schools can thus be summarized in six selection channels. Selection channels {A1 B C}, {A2 B C}, and {A3 B C} include 29 students, 13 of which entered their first choice school (A1, A2, or A3), while the others faced a 11

12 randomization on their second choice. Only 8 of them obtain school B and the 8 others ended up in school C. The other three channels {A1 0 C}, {A2 0 C}, and {A3 0 C} include 183 students, of which 85 entered their first choice school. All others selected either C or an A school in their second choice, and thus were all assigned to C. We summarize all possible types of channels encountered in the whole district in Table 2, and show where the children were enrolled. The 163 students that are under a selection channel of type {A 0 0} selected only schools of type A in their choices, and thus were enrolled in a C school if they missed their first choice. By far the most frequent channel type is {A 0 C}, corresponding in most cases to a sequence of invalid choices (school already filled in earlier rounds) before the choice of a C school. The last two channel types correspond to all the choices that led to a second randomization process. As currently specified, some channels are common to neighborhoods that share access to the same schools. To further control for unobservable variables reflected in the school choices, we thus define a selection channel as being a selection sequence in a given neighborhood. There are 137 such channels, with each neighborhood having between 3 and 31 of them. These channels perfectly characterize all the selection processes that affected the placement of children in schools other than the random drawing Test of validity of the randomization In this section, we test for the validity of the randomization process. We only consider the 4350 students that were identified as enrolled through the random assignment process described in the previous section. In principle, the average exogenous characteristics should not differ systematically among students that are randomly selected in or out of the school they have chosen. We first verify the validity of the randomization process for each of the 12 A schools that selected their students on their first choice (step 1 randomization). Since some of these schools serve several 12

13 neighborhoods and have separate quotas and randomization per neighborhood, we need to control for the neighborhood. We thus estimate the following regression: x in = α + δ IN i + η n + ε in for each school where x in is a characteristic of student i from neighborhood n, IN i is equal to 1 if the student is selected in the chosen school during the random assignment, and η n is a neighborhood n fixed effect. Results are reported in Table 3 for two student characteristics (gender and primary school test score) and several parents characteristics (income, education level, two socio-economic indicators newly developed in China that rank people according to their profession and rank, and whether the parents have a relative in the school). We also analyze four indices of parents attitude toward school and their children: parents expectation on the academic achievement of their child, the importance that they attached to the school quality on child s growth according to their responses to a census question (on a scale of 1 to 5), an index of their involvement in their child study and life (an average of the response to three questions, each on a scale of 1 to 5), and an index of their knowledge of their child s life at school (a similar average of responses to 3 questions, each on a scale of 1 to 5). The results show almost no significant difference in child s characteristics. Important parent characteristics (income, education, and socio economic ranks) are, however, significantly higher among children that were accepted for 5 or 6 out of the 12 schools. More specifically, income is substantially higher and significant in two schools. While parents education is significantly higher in 6 schools, the difference is small for 2 of them. These significant differences raise some questions. It may be the case that with some relatively small pools of children (at the school-neighborhood level), a few outliers would raise the average even in a fair randomization. We therefore pool all the schools and perform an overall test, with neighborhood-school fixed effects: x ins = α + δ IN i + η ns + ε ins 13

14 where the index s indicates the school to which the student applied. The results are reported in Table 4. In column 1, the full sample of 4350 students are pooled together, and in column 2, we restrict the test to the neighborhood-school pools that have at least 140 applicants. While parents characteristics remain significant when we pool all the students together, they are much weakened in significance and in size when the sample is restricted to large neighborhood-school pools. Results for pooled regressions for the step 2 randomization are reported in column 3. With a total of only 556 students randomized at this level, it is not possible to do individual school estimation. No significant difference between children appears in this step 2 randomization process except for parents knowledge of their child s study, life and peer group, one profession-based social-economic index, and whether they have acquaintance in the school or not. However, the difference is relatively small, especially considering that the group size should be quite small given that there are only 556 students in total randomized at step 2. The coefficient of relative in the school is significantly negative, indicating that having a relative in the school will decrease the chance of getting in, which is not intuitive. In the analysis of students performance, we will compare students that belong to the same selection channel, arguing that the school to which they have been assigned within each channel is random. Critical for the validity of that analysis is, therefore, the verification that children randomly selected in or out of a school within a channel are similar. We thus repeat similar tests of difference in students and parents characteristics, controlling for the selection channel: x ic = α + δ IN i + η c + ε ic where x ic is a characteristics of student i from channel c, and η c indicates the effect of channel c. Results are reported in Table 5. There is no significant difference in these variables between students randomly selected in the step 2 random assignment and those randomly selected out; however, for students who are randomly selected in the first step randomization, parents opinion of 14

15 school importance, one of the profession-based socio-economic status indices, parents income and education, and indicator of having acquaintance in the school are significant. In particular, the difference in parents income is higher than 10% of the average income. Considering the fact that many channels have very small observation numbers, and thus these results could be driven by some abnormalities of small sample size, we do the same tests restricting the sample to channels with more than 90 observations. Restricting the sample to channels with more than 90 observations, only one index of profession-based socio-economic status of parents, indicator of acquaintance in the school and parent s education turn out to be statistically significant at the 10% level. However, the difference in the socio-economic status is very small in scale, less than 5% of the average level. As to the indicator of acquaintance in the school, only very few people (less than 3% of the total responses) in the data set actually have an acquaintance in the school, thus it is unlikely that this difference will affect the overall results of school effects significantly. In conclusion, there are some significant differences in characteristics between those randomly selected in and out during each step of the random assignment (mostly in the first step random assignment), indicating possible under-reporting of enrollment without random assignment and necessity for extra control later to check the robustness of a school effect. However, most of the significant differences in characteristics seem to be driven by abnormalities due to a small number of observations in some channels and are not consistent when the sample is restricted to large channels. The difference does not appear to be strong enough to compromise the fundamental randomization design and affect the overall results regarding the impact of school quality. Furthermore, transfer after the randomization is very restricted, and could only be realized through considerable financial contribution to the middle schools. And there should be very few slots left after the school had accommodated all students coming through random assignment or directly enrollment. Thus, there should not be a significant under reporting of the transfer students. As a result, although the characteristics of these few students, for example, parents income, will make a significant difference, esp. when 15

16 the comparison group is small, they are less likely to generate an effect of similar size in later school effect analysis. 4. Results of School Effects We would assume that the popularity of a school, as revealed by the choices expressed in application forms, largely reflects the achievement of the school in terms of academic performance. This can be verified by looking at the High School Entrance Exam test scores before the reform. In Figure 2, the horizontal axis reports the overall school rank on that exam, and we can verify that all the most popular A schools (represented with a * symbol) have the lowest rank, and that all but two of the B schools have lower rank than C schools. This indicates that school performances were well known to parents and considered important when they made their application decisions. Before we analyze this issue with individual scores, it is instructive to look at the changes in relative performance after the reform. Figure 2 shows a clear trend in the change in ranking, with worse performing schools (high rank) all gaining between 1999 and 2002, and better schools loosing ranks. More specifically, the top 5 schools did keep their very high rank, and the equalizing trend took place among the other schools. Remarkably 3 of the least performing schools and 2 intermediate schools moved up to the top 10 schools, replacing 5 A schools that dramatically fell to the lowest level. This pattern can be observed for all five subjects included in the High School Entrance Exam (Chinese, mathematics, physics, chemistry, and English), although the equalizing factor, measured by the correlation between rank in 1999 and change in rank, is highest for Chinese and mathematics. There might be several reasons for this change. First, the random assignment procedure is likely to have equalized the quality of students across schools. Second, teachers in the best schools, who have been used to teaching good students, may lack the experience and expertise to teach less qualified students. While admittedly good students are considered to be easier to teach than less qualified students, common 16

17 wisdom in the school district is that teachers in the best schools are in general superior in subject knowledge and lecturing skill but less competent in student management and stimulating students interest than their colleagues in the less coveted schools. As a result, while teachers in formerly low ranking schools are faced with an easier task due to the improvement in students quality, teachers in the most popular schools are faced with a more difficult task in tutoring students who are less motivated, and possibly are less talented than those they used to teach. This may contribute to the decreasing effectiveness of previously best schools. Finally, after the reform, the formally best schools no longer received privileged access to public financial appropriation and priority in selecting new teachers. Table 6 gives three-year average levels of school resources in terms of teacher s rank, experience and education, etc. From this Table, we can see that previously top schools are not that superior in teachers quality. To formally examine the school effects on student performance, we limit the analysis to students that were randomly assigned to schools, i.e., the 4350 students in Figure 1. Also, due to missing information in students test scores, the observation number drops to First, we want to look at whether school effects exist or not. To do this, we regress students test scores in all subjects on individual school dummies, controlling for the selection channel dummies. Because students in the same channel are randomly assigned to different schools, the school assignment should be orthogonal to unobserved students characteristics for students in the same channel. Thus, after controlling the channel effects, the coefficients of the school dummies should be unbiased estimates of the overall school effects averaged across selection channels. The regression model is: yics = α + η + γ + ε s c ics where y ics is the score obtained by child i, from selection channel c, enrolled in school s. γ c is the effect of selection channel c. The coefficient η s denotes the school effect of school s. Summary results are reported in Table 7. They show that school effects are 17

18 important in determining test score in all subjects since all F-tests of the joint significance of school dummies have p value less than Because some individual variables are significant in the test of random assignment when the whole sample is used, which raises some concern of under reporting nonrandom enrollment and thus confounding individual characteristics, it is important to see whether including some individual characteristics will change the coefficient estimates significantly or not. Thus I run the same regression controlling for both the selection channels and individual characteristics including parents income, parents education level, student s primary school test scores, and student s gender. With joint school effects turning to be weakly significant in F-test of overall score, school effects are significant in all subjects at the 0.01 level. However, the F-test can only show that school effects matter, yet we cannot see whether school effects matter or not in the expected directions. In another word, it doesn t show whether schools with stronger positive effects tend to have better performance in relevant exams or not, and vice versa. Thus, we further check the correlations between individual school effects and school ranks in overall score and all subjects in As shown in Table 7, correlations between individual school effects and school ranks (in overall score as well as in all subjects) in 2002 are significantly negative, i.e. the school effects are significantly positively correlated to school performances in Exam On the contrary, correlations between individual school effects and school ranks in test scores in overall score and all subjects in 1999 are relatively low and insignificant. These results thus not only confirm the importance of school effect, but also show that much of the ranking of schools prior to the reform was the outcome of student selection rather than school effects. Finally, we do the same regression, except using school type effects rather than individual school. The resulting school type effects shown in the last section of Table 7 have mixed signs and are insignificant in most cases. It shows no obvious superiority of the previously better A schools over the other schools, which also indicates that ranking in performance of schools prior to the reform was dominated by student selection rather than by authentic school effects. 18

19 5. Effects of Specific Teaching Resources on Students Test Scores In this section, instead of just examining the school effects, I will explore the effects of school resources, more precisely, teaching resources, on students test scores, conditional on students randomization channels. I consider the school resources in terms of teacher-student ratio, school-average teacher s rank (in China, this is an official evaluation of a teacher s comprehensive quality and performance, and a teacher s welfare depends on his rank), formal education, informal training, experience, and gender ratio. I consider the school-average level of teacher s characteristics instead of matching teachers with the students they teach because I cannot completely exclude the possibility that schools may assign teachers to each class according to students characteristics in the relevant class, thus biasing the results. Using the school-average level (or junior-school average level) of these characteristics can help avoid the problem of class level selection bias, though it is less precise than the matching case. Thus, the formula used is: y = α + β + γ + ε ics x s c ics where y ics is the High School Entrance Test Score in the relevant subject of student i, who belonged to channel c and finally enrolled in School s and x s is the teachers characteristics of school s, which i finally enrolled in. γ c is the effect of channel c, which student i belonged to. In addition to missing information in students test scores, four of the schools were merged to other middle schools and thus I don t have complete teachers information available for them. As a result, the sample only includes 1981 students out of the original 4350 students enrolling through the random assignment process. Since students choose which channel to enter through their applications and then, in each channel, are randomly assigned to different schools, school resources should be orthogonal to the observed or unobserved individual characteristics after controlling for the channel effects. Thus, the coefficient estimates should reveal the true average effects 19

20 (across different channels) of corresponding school resources on students test scores. The results are summarized in Table 8. Teacher s rank is significantly positive in all subjects at least at the 0.01 level, and significantly positive at the 0.05 level in total score. As aforementioned, teacher s rank is an official evaluation of teacher s quality. In China, to decide a teacher s rank, the education bureau not only considers some objective measures of the teacher s capabilities such as formal education level, training, experience, and honors, but also collects subjective information such as opinions from headmasters, the teacher s colleagues, students, and parents. They also send officials to audit the teacher s lectures and give grades. Because the teacher s position, salary, and welfare, etc. are strongly dependent on their rank, the rank evaluation process is usually very systematic, rigorous, and well organized. Thus, the resulting teacher s rank should be a reliable comprehensive indicator of a teacher s quality. Positive significance of teacher s rank provides positive evidence for the effectiveness of teacher s quality, and perhaps indicates that the current teacher evaluation system is functioning reasonably well. Teacher s formal education is only significantly positive at least at the 0.1 level for Chinese. It seems that teacher s rank has captured the major effect of a teacher s formal education level, so that there is not much additional significance in a teacher s formal education level after considering his rank. Both teacher s informal education and experience have, in most cases, significant negative coefficients. This result persists when I use different measures of teacher s experience (years of teaching, years of teaching graduating classes, and years of teaching as head-teacher). It indicates that the beneficial effects, if there are any, in teacher s experience and informal trainings might have been captured by the effect of teacher s rank. The fact that the additional effects of teacher s informal training and experience are negative is instructive. It might indicate that the various on-going teacher s additional informal professional training (many of which are mandatory and unsystematic), especially after the education reform took place, might be more like a disturbance in teachers regular work than a plus in their teaching skills. Thus, the system of informal professional training needs to be improved. The negative coefficient of teacher s experience may indicate that long-term 20

21 teaching may not be beneficial to a teacher s overall skill and engagement in teaching as many previous studies have suggested. One possible reason for this is that, in Beijing, middle school teachers are generally over-loaded with work, and thus long-term teaching may decrease their productivity as a result of job burn out. Another possible reason is that, as the teacher gets older, his teaching style and methods may become outdated. And if older teachers fail to update their teaching strategy as the school situation changes, they may become less productive than their younger colleagues. Teacher s gender ratio doesn t show any significance. For the other two additional measures, teacher-student ratio has large coefficients but is not significant. Teacher s gender-ratio is significant in Math, Chemistry, and English. However, it is difficult to give any specific interpretation without more information regarding school characteristics because these measures are related to too many aspects of school conditions such as funding and tradition. Nonetheless, it is important to keep these two indicators in the regression to secure unbiased estimates for the major factors such as teacher s rank. To check whether the results in Table 8 are driven by abnormalities due to the small number of observations or not, I restrict the sample to channels with no less than 90 observations, i.e., the sample with which I validate the random procedure using channel effects as control. Results in Table 9 show that, by doing so, there is no conflicting evidence coming up regarding the major factors of interest, i.e., teacher s rank, informal training, and experience. Moreover, although I have argued in a previous section that evidence of difference in individual characteristics among students who are randomly selected in and those randomly selected out in each step of random assignment is not strong enough to invalidate the randomization procedure, I put in some important individual characteristics as control variables to double check whether results regarding school teaching resources effects are robust. Results are summarized in Table 10 to Table 12, with the first column of each subject showing coefficient estimates without control and their 95% confidence intervals, and the second column showing results after controlling parents average income, education, student s gender, 21

22 and test scores in primary school examination. As shown in the second column of each subject, there is no essential change in results regarding the effects of the major indicators of teacher s quality such as teacher s rank and experience, even when the sample size decreases due to limited availability of individual characteristics data. Even if the randomization procedure is valid and thus students individual characteristics will not compromise the validity of the coefficient estimates of the teacher s characteristics, the results above may not authentically reflect the importance of school quality (to be more precise, the importance of the teacher s effect). A significant proportion (2752 out of 7012) of the whole students body doesn t go through the randomization procedure. Instead, these students choose the schools they enroll in and avoid being randomly assigned. Why do these students choose that particular school? There might be something in the school spirit or other unobserved characteristics matching them best. Although I cannot identify what those factors exactly are, they are probably reflected in the characteristics of the students who choose and manage to enter that school. And once these students enroll in that school, they also constitute an external atmosphere to the randomized students, and this atmosphere is also randomly assigned to the randomized students, and may affect their academic performance. Thus, the results of the teacher s effects may also capture the effects of school atmosphere. For this reason, I add the average characteristics of the nonrandomized students for each school into the original regression as control variables, in addition to students individual characteristics. I index this factor as school atmosphere, the nonrandomized students as peer to the randomized ones, and use four variables, namely, the peer s parents education, the peer s parents income, the peer s primary school test scores, and peer s gender ratio as proxies to the school atmosphere. The results are shown in the third column of Table 10 to Table 12 for each subject. Thus for each subject in Table 10 to 12, the first column includes results without control variables, and the figures in parentheses are confidence intervals of the coefficient estimates. The second column includes results when controlling individual 22