GSA & HMC Synopsis of the Report on the 2015 Cambridge International Examinations (CIE) IGCSE First Language English (0500) results in GSA & HMC schools Dr Peter Mason April 2016
The problem In summer 2015, HMC reported 1 that over 80 of its member schools had serious concerns about their results in Cambridge International Examinations (CIE) IGCSE English 0500. This is an unprecedented number of school responses. In the light of the extent of this unease, and following face to face meetings with CIE staff, GSA & HMC commissioned a detailed report to receive and analyse data from schools in order to answer the questions: was there really a problem? if yes, how did it arise and what was the extent of the problem? if yes, what has been CIE s response to the problem? If yes, what, if anything, can be done to protect the interests of the 2015 candidates and to ensure that there is no repetition of the problem in 2016? This is a synopsis of the report s findings. CIE offer two IGCSE English syllabuses denoted as 0500 and 0522. The syllabuses are closely linked with some common examination elements. Syllabus 0500 is intended for overseas centres and UK independent schools; it is not accredited by Ofqual for use by maintained schools in England whereas Syllabus 0522 is accredited by Ofqual for use by maintained schools. In recent years, growth in the number of June entries has been modest for syllabus 0500 reaching only around 17,500 candidates in June 2015. Schools in membership of GSA and HMC provided a significant proportion of these candidates. By contrast syllabus 0522 has seen extraordinary growth reaching close to 195,000 entries in June 2015. Such dramatic growth presented its own problems to CIE particularly in recruiting, training and monitoring additional examiners and setting the grade standards for a vastly changed candidature. GSA and HMC schools did not report any major or widespread problems with the June 2014 0500 examination, nor has any evidence been received that schools that entered candidates for syllabus 0522 in June 2014 were dissatisfied with the results. This unprecedented level of concern focuses on the June 2015 examinations only. CIE report that, in May 2015, as part of their post results analysis, they judged the 2014 results, in both IGCSE English syllabuses to have been too lenient. It is not clear what prompted this review so long after the event, nor is it clear which other results CIE compared their own with, so the stated basis for this decision appears obscure and not wholly convincing. Ofqual report that they had been monitoring CIE s IGCSE English for some time so there are bound to be suspicions that CIE s review of its standards was due to pressure from Ofqual. Whatever, the decision was taken to tighten standards in the June 2015 examinations at grades A, C and E, a decision that was not conveyed to schools though CIE assert that Ofqual was kept informed. The outcome was widespread alarm among schools when results (for both syllabuses) were published in August 2015. GSA, HMC and ASCL all conveyed a high level of concern to CIE. In response CIE published two general briefing documents and a technical document for schools and headteacher associations in an attempt to explain how they had set the standard for the June 2015 examinations. In these there was no admission from CIE that the standard was wrong though setting the standard at key grade boundaries must have been exceptionally difficult given the inextricable link between the two syllabuses with common components: papers 1, 2, 3 and the coursework (paper 4) all of which must have a common standard between the two syllabuses. Further, even 1 HMC briefing note to members 05 November 2015 SYNOPSIS (v2) of GSA & HMC report: 2015 CIE IGCSE English (0500) April 2016 page 1
within each syllabus the different papers provide different pathways through the assessment with the coursework being an alternative to the written paper 3. Also, grade C is common between core and extended tiers so the grade C standard must be aligned between these two sets of candidates. And finally amid all this complexity, add the huge rise in new candidature for syllabus 0522, a large proportion of whom (according to CIE) were C/D candidates and the challenge of reliable standard setting becomes exceptionally difficult. The evidence GSA and HMC took the understandable view that a full analysis of schools CIE 0500 results must be undertaken with a detailed report to better understand the origins and extent of the problem. This was made more straightforward since all the affected schools entered candidates only for the extended tier covering grades A*-E. Within this range concerns centred largely on the top grades i.e. grades A* and A. 53 GSA & HMC schools covering almost 5,000 candidates submitted a range of relevant detailed data that enabled an in-depth analysis to be undertaken. In particular they provided raw marks for each candidate on each component taken syllabus (qualification) grades and grades on each paper any changes to the above following EARs the schools own estimated syllabus grades for each candidate Qualification grades were generated simply by summing the raw marks on each paper giving a maximum score of 100. For CIE 0500, extended tier candidates were entered for papers 2 and 3 and the alternative route of paper 2 and coursework component, paper 4. The 2/4 combination was taken by the majority of candidates (3,701) and produced a grade profile well below expectations; the 2/3 candidates (1,202) were also affected but to a lesser extent. Syllabus grade distributions Papers/components A* A*/A A*-B A*-C A*-D A*-E A*-U entry 2/3 candidates 42.3% 69.1% 88.4% 97.8% 99.5% 99.8% 100% 1,202 2/4 candidates 16.3% 50.6% 81.2% 95.2% 99.5% 99.9% 100% 3,701 There is clearly a problem, though the huge disparity, (especially at grades A* and A), between the two separate entries might suggest that the problem is solely that of the 2/4 candidates. But that is not the case; schools that entered candidates for papers 2 & 3 also expressed serious concern at their candidates outcomes. Closer scrutiny of paper 2 showed a paper grade profile completely out of line with the abilities of the candidates. Schools found that many of their high achieving students gained C, D, E or even U grades on this paper giving rise to considerable and understandable alarm. The problem encountered by schools could be attributed, in the main, to this paper. CIE revealed in its technical paper that the mark range between unit grade A and unit grade C on paper 2 was just 4 raw marks. Schools inevitably questioned whether the marking of an English paper could ensure the accuracy and precision required, especially with the large number of additional examiners recruited to mark the additional entries. SYNOPSIS (v2) of GSA & HMC report: 2015 CIE IGCSE English (0500) April 2016 page 2
Since paper 2 was common to both the 2/3 candidates and the 2/4 candidates, did the disparity shown above arise due to inherent differences in the abilities of the two sets of candidates? Comparison of how both sets performed on paper 2 shows a remarkably similar profile. Paper 2 achievements A* A*/A A*-B A*-C A*-D A*-E A*-U 2/3 candidates 32.8% 47.5% 61.2% 75.3% 89.3% 93.6% 100% 2/4 candidates 33.2% 48.2% 63.1% 76.1% 90.6% 95.1% 100% [the A* grade is strictly a qualification grade and is notional on individual units. However, it can be determined by extrapolating the mark range above the grade A boundary by the A-B mark range interval] When compared to the data in the table at the foot of this page, the fact that only around three quarters of all candidates were judged to have achieved grade C or better on this paper confirms that there has been a problem. On paper 3 the grade profile was more in line with expectations and partially compensated for the impact of paper 2. However paper 4 (coursework) exacerbated the problem at A* and A grades. Papers/components A* A*/A A*-B A*-C A*-D A*-E A*-U 3 (written) 50.6% 78.1% 95.6% 99.4% 99.7% 99.9% 100% 4 (coursework) 5.8% 62.7% 91.1% 99.4% 100% 100% 100% Well-taught and well-motivated students such as those in the schools in this analysis would be expected to achieve very highly on the coursework component of any examination. The A* percentage (5.8%) and cumulative A*/A percentage (62.7%) are both well below what would normally be expected. When schools are concerned about examination results they usually carry out two immediate comparisons: (i) with their forecast (estimated) grades and (ii) with the candidates grades in all other subjects and in particular related subjects. (i) Comparison with estimates Grade profile A* A*/A A*-B A*-C A*-D A*-E A*-U Av subject pts score Actual (cum %) 24.2% 58.0% 85.2% 97.3% 99.8% 99.9% 100% 6.64 Estimated (cum %) 37.1% 71.7% 93.5% 99.6% 100% 100% 100% 7.02 (ii) Comparison with all other subjects and with the related subjects English Literature and History A* A*/A A*-B A*-C A*-D A*-E ENTRY Av subject pts score English IGCSE (CIE 0500) All subjects except English English literature 27.4% 61.0% 86.6% 97.5% 99.8% 100% 4836 6.72 48.7% 76.2% 91.6% 97.8% 99.5% 99.9% 43293 7.14 49.7% 76.9% 92.8% 98.5% 99.7% 100% 4776 7.18 History 49.9% 79.9% 94.4% 98.3% 99.5% 99.9% 2915 7.22 SYNOPSIS (v2) of GSA & HMC report: 2015 CIE IGCSE English (0500) April 2016 page 3
This in-depth scrutiny shows that candidates achieved far less well in IGCSE English 0500 than in their other subjects combined (on average) and in English literature and in history in particular, two subjects which require similar skills to English. To counter the argument that this might be a national trend, or a feature of results obtained by more able students, further equivalent detailed comparisons were made with all GCSE candidates nationally, and candidates from a basket of 25 selective maintained (grammar) schools. These also showed that the IGCSE English 0500 candidates had achieved much less well than would have been expected. The volume of data and the analyses speak for themselves. It is an inescapable conclusion that a major problem existed with the June 2015 CIE 0500 results in a large number of GSA & HMC schools. The question remains of why the problem wasn t identified before results were finalised. CIE s own Code of Practice provides for a number of checks that must be made before results are released, all of which are designed to minimise error. Schools have seen little evidence that these checks were actually done. Unsurprisingly there was a huge increase nationally in enquiries about results (EARs) in 2015 compared with 2014 for both syllabuses, though 0500 (87% increase) was small in comparison with syllabus 0522 which saw a 225% increase. For the schools in this study the proportion of EARs that resulted in grade changes was 16.8%, greater than the CIE national figure of 8.6% though both are it is lower than the figure Ofqual reports (18.0%) for GCSE nationally. A significant number of schools remained dissatisfied with EARs and therefore lodged appeals but no school s appeal (Stage 1 or Stage 2) had been upheld at the date of releasing this report despite schools raising non-compliance with the Code of Practice as part of their case. The conclusion In summary, despite the overwhelming data-rich evidence that there was a major problem with the June 2015 CIE 0500 examination, CIE maintain that the standards they set were correct. It is recommended that GSA and HMC further engage with CIE and possibly with Ofqual about the evidence in this report with the aim of exposing the problems that arose in the CIE 2015 IGCSE English (0500) examination and achieving justice for the 2015 candidates. GSA and HMC will also want to ensure that there is no repetition for candidates in June 2016. Dr Peter Mason April 2016 SYNOPSIS (v2) of GSA & HMC report: 2015 CIE IGCSE English (0500) April 2016 page 4