Insight: Schools Scenarios Introduction: These scenarios are designed to familiarise users with Insight by providing an analysis of data presented through national and local measures and their accompanying filters. In addition, scenarios offer evaluative questions that can be considered alongside this data. They approach the use of Insight from the perspective of users in local authorities and schools. This is not intended to be a full analysis but rather gives an example or indication of some paths of enquiry that could be followed and pertinent questions to be asked. Taking a look at your school Data from Insight will be used by your school to understand how its performance compares against its virtual comparators and the national picture (Note: an explanation of virtual comparators and how they are calculated can be found at: http://insightguides.scotxed.net/technical.htm#virtual_comparators). Schools can use the tool to analyse performance within curricular areas and examine how this compares against benchmark data. This will help you as a school to arrive at an understanding of how well you are meeting the needs of all learners and to identify areas for improvement. The following scenarios focus on a few of the national measures as well as local measures looking at a particular curricular area. They do so from the perspective of a whole school, but can also be used by all stakeholders to familiarise themselves with key aspects of the tool. National and local measures are connected and therefore should be used together in considering performance and the next steps to be taken to seek further improvement. Scenarios include: 1. Introduction to whole school level data and analysis 2. Exploring national measures together at whole school level 3. Exploring a curricular area at school level 1. Introduction to whole school level data and analysis Let s begin by considering how one measure might prompt us to ask questions that develop our understanding of performance in the senior phase. Improving Attainment in Literacy and Numeracy This measure on the national dashboard looks at attainment in both literacy and numeracy. The measure focuses on: The proportion of leavers achieving literacy and numeracy at SCQF (Scottish Credit and Qualifications Framework) Level 4 and above and SCQF Level 5 and above. [Note: More information on the SCQF can be found at: http://www.scqf.org.uk/] and a briefing paper on literacy and numeracy qualifications can be found at http://www.scotland.gov.uk/topics/education/schools/curriculum/seniorphasebenchmarking/ supportingmaterial/literacyandnumeracybreifingpaper 1
The graph below provides data on attainment in literacy and numeracy for leavers in a school over a three year period and is updated annually to reflect the attainment of those leavers in the previous academic year (please note that the graphs in the first sections show the colour scheme found in the preview editions, which was updated for the live version) : In tabular form, this graph converts to: Name Year % Level 4 Literacy and Numeracy % Level 5 Literacy and Numeracy School 2011 79.17 43.75 Virtual Comparator 2011 81.38 54.04 School 2012 84.74 47.89 Virtual Comparator 2012 82.89 55.53 School 2013 83.66 47.03 Virtual Comparator 2013 81.56 54.33 2
Discussion: An initial look at such information should prompt us to ask some straightforward preliminary questions, such as: What does this evidence illustrate about the attainment of the school s leavers at SCQF Level 4 and above in literacy and numeracy? What does this evidence illustrate about the attainment of the school s leavers at SCQF Level 5 and above in literacy and numeracy? What can we learn from the data, when comparing the school s performance to that of its virtual comparator? Why is the percentage of leavers attaining SCQF Level 5 and above in literacy and numeracy lower than that of the virtual comparator? By making use of the measures positioned directly below the graph, it is possible to explore this information further and use it as the basis of additional questioning: Click here to see this information by SCQF level Click here to see this information by measure For example, by accessing the data by discrete measure, the percentage of the school s leavers attaining literacy at SCQF Level 4 and above and SCQF Level 5 and above can be seen as: 3
Establishment Year % Level 4 Literacy % Level 5 Literacy School 2011 99.48 67.19 Virtual Comparator 2011 95.31 67.84 School 2012 98.95 75.79 Virtual Comparator 2012 96.32 73.95 School 2013 98.51 67.82 Virtual Comparator 2013 94.43 69.68 In addition, the percentage of the school s leavers attaining numeracy at SCQF Level 4 and above and SCQF Level 5 and above is: 4
Establishment Year % Level 4 Numeracy % Level 5 Numeracy School 2011 79.17 45.83 Virtual Comparator 2011 81.90 57.29 School 2012 84.78 48.95 Virtual Comparator 2012 83.42 57.63 School 2013 84.16 50.00 Virtual Comparator 2013 82.55 57.05 Discussion: Having explored the information in this way through the use of an additional filter, it is appropriate to apply similar questions as before: What does this evidence illustrate about the attainment over time of the school s leavers at SCQF Level 4 and above in literacy as a discrete measure? What does this evidence illustrate about the attainment over time of the school s leavers at SCQF Level 4 and above in numeracy as a discrete measure? What does this evidence illustrate about the attainment over time of the school s leavers at SCQF Level 5 and above in literacy as a discrete measures? What does this evidence illustrate about the attainment over time of the school s leavers at SCQF Level 5 and above in numeracy as a discrete measures? What can we learn from the data when comparing the school s performance in literacy and numeracy to that of its virtual school? In particular, drilling down into the data should help address the question: Why is the percentage of leavers attaining SCQF Level 5 and above in literacy and numeracy, lower than that of the virtual comparator? Using these key filters in this way will allow schools to identify areas for improvement and help them reflect on appropriate points for action to be carried out in light of any issues that might emerge. Moreover, as a further means of answering such questions, dashboard filters are available below the graphs to help develop our understanding of any emerging trends in performance. These filters are: 5
For instance, if we wish to consider separately the attainment of different groups of learners at SCQF Level 4 and above and SCQF Level 5 and above in numeracy, the Gender filter could be used to identify the following: Male leavers: 6
Female leavers: Male leavers: % SCQF Level 4 and above 2011 2012 2013 School 76.77 89.25 80.77 Virtual Comparator 75.25 81.72 81.25 Male leavers: % SCQF Level 5 and above 2011 2012 2013 School 41.41 52.69 49.04 Virtual Comparator 48.99 57.53 50.96 Female leavers: % SCQF Level 4 and above 2011 2012 2013 School 81.72 80.41 87.76 Virtual Comparator 88.98 85.05 83.93 Female leavers: % SCQF Level 5 and above 2011 2012 2013 School 50.54 45.36 51.02 Virtual Comparator 66.13 57.73 63.52 7
Discussion: Deploying such filters allows us to see which groups of learners are in line with or are performing better than their virtual comparators, as well as where improvements in performance could be made. Consequently, we might ask: Can we identify areas for improvement in relation to a particular group of learners? Such an analysis is carried out in the context of fundamental questions such as: Are we meeting the needs of all our learners? What specific action needs to be taken to improve the performance of any particular group? How do we ensure that the attainment of our learners continues to move in a positive direction? Filters could be deployed further to explore such lines of inquiry by examining the performance of male/female leavers in literacy against the performance of male/female leavers in numeracy, including comparing to the corresponding virtual comparator in each case. This would enable the following to be asked: How does the boys literacy compare to the boys numeracy in each SCQF level? How does the girls literacy compare to the girls numeracy in each SCQF level? 2. Exploring national measures together at whole school level It is important to note that consideration of leavers performance should be carried out through a combined analysis of measures on the national dashboard. Such an approach will help users gain as full a picture as possible of their school s progress against the range of measures. For example, any analysis of leavers performance in literacy and numeracy, such as that above, should be carried out in conjunction with a similar analysis of the other measures on the national dashboard. Let s now take a look at some of these. When compared against its virtual comparator, local authority and national statistics, it can be seen that this school s percentage of leavers in a positive destination has fluctuated slightly over a three year period: 8
2010/11 2011/12 2012/13 Establishment Year % of School Leavers in a Positive Destination School 2010/11 85.94 Virtual Comparator 2010/11 85.81 Local Authority 2010/11 78.20 National 2010/11 81.54 School 2011/12 80.53 Virtual Comparator 2011/12 87.63 Local Authority 2011/12 81.04 National 2011/12 83.85 School 2012/13 91.09 9
Virtual Comparator 2012/13 83.91 Local Authority 2012/13 82.77 National 2012/13 81.14 Through the use of the various filters, questions that could be asked in order to analyse this data further include: How do the girls positive destinations compare to those of the boys? Does an analysis of particular groups of leavers (through the use of filters such as Stage of leaving) reveal any positive or negative patterns that need to be explored further? In addition, an analysis of the Improving Attainment for All measure shows that the school is in line with or slightly better than its virtual comparator in this measure: Lowest 20% Middle 60% Highest 20% School 182.23 747.78 1546.76 Virtual Comparator 156.52 749.83 1538.25 Local Authority 110.01 587.49 1504.92 10
National 128.82 607.01 1412.19 [Note: An aspect of this national measure average total tariff score allows for the comparison of attainment between groups of pupils, schools, local authorities or other cohorts. Each qualification (unit and course) included in the Scottish Credit and Qualifications Framework (SCQF) is awarded tariff points based on the SCQF level and the SCQF credit points of the qualification. This allows any qualification that is SCQF levelled to be included in the tariff score in an open and transparent way. Further information can be found at: http://insight-guides.scotxed.net/technical.htm#tariff ]. Key questions to be asked when considering the comparative data offered by this and the previous measure are: How well are we meeting the needs of, and improving attainment for, all learners within our school i.e. the lowest 20%, Middle 60% and Highest 20% of attainers? Is there a link between the percentage of leavers in a positive destination over a three year period and changes in attainment (or might this be due to other factors)? Moreover, by deploying the various filters available through the tool, additional questions might include - Are we meeting the needs of: the highest attaining girls? the highest attaining boys? the lowest attaining girls? the lowest attaining boys? ASN pupils? those pupils for whom English is an additional language (EAL)? Doing so will help schools detect any issues in aspects of the attainment data illustrated through this measure. In this way, it is possible to refine an analysis of particular groups of leavers in order to determine how they fare in comparison to their peers and against the virtual comparator as well as local and national data. Attainment versus Deprivation So far, we have concluded that the school is generally in line with, or above, its virtual comparators in two of the national measures increasing post school destinations and improving attainment for all. Let s now compare what we have established about the school s attainment with another of the national measures which tackles disadvantage by focusing on improving the attainment of lower attainers relative to higher attainers: 11
Within this measure we can analyse the performance of the school s leavers in light of the Scottish Index of Multiple Deprivation which goes from the 10% most deprived areas (1) to the 10% least deprived areas in the country (10). The red discs show the attainment of pupils in each decile nationally and are of roughly equal size, revealing an even spread across all ten deciles. The graph also illustrates the correlation between deprivation and attainment, with those leavers in the upper deciles tending to attain higher than those in the lower deciles. The vertical axis shows the average total tariff score while the orange discs illustrate the situation in a particular school. When considering the average total tariff score of pupils in your school against the SIMD, there are two main things to consider: The size of the orange discs across the deciles, showing the backgrounds of your pupils The comparison of each orange disc to the national red disc in each decile, indicating the performance of pupils in that decile compared to the national picture. Discussion: In the graph above: How does the school (orange) compare against the national data in each of the SIMD deciles (red)? Are there particular features about the school s performance that require further exploration in this measure? 12
In answering the second question, you may wish to consider the following outline of how the virtual comparator measure - used in the increasing post school destinations and improving attainment for all national measures already looked at - is calculated to include the Scottish Index of Multiple Deprivation (SIMD): The virtual comparator is made up of pupils from schools in other local authorities who have similar characteristics to the pupils in your school or local authority. It allows you to see how the performance of your pupils compares to a similar group of pupils from across Scotland for the purposes of self-evaluation and improvement activities. For each pupil in the cohort of interest (e.g. S4 pupils in School A), 10 matching pupils are randomly selected without replacement from other local authorities based on the following characteristics: Gender Additional support needs latest stage Latest stage Scottish Index of Multiple Deprivation. In fact, the graph indicates that a large number of pupils in the school are in band 9 of the SIMD and that, in all but one bands, the school is below the national trend; significantly so for band 9 and others. Given the definition of the virtual comparator (based, in part, on the SIMD), this evidence seems at first glance to be at variance with our analysis of the school s data in the other two measures, which suggests that the school is mostly in line with its virtual comparators. Consequently, this prompts us to dig deeper in order to reconcile these different assessments of the school s attainment. Doing so, should help determine possible areas for improvement. As can be seen from the description of the SIMD above, one thing that the virtual comparator controls for that the SIMD measure does not is Additional Support Needs. It may be that the school has a particularly large number of ASN pupils, which would have an overall impact on attainment on the SIMD measure but also have an impact on the attainment of the virtual comparator school. Consequently, this could account for the variation. Again, the filters provided below each of the national measures can be used to explore the data presented from a variety of perspectives. For example, by selecting Other in the ASN filter, it is possible to select the SIMD measure to only show non-asn pupils. If the difference were due to the proportion of ASN pupils, then we would expect the comparison between the school and the national trend to have improved: 13
However, in doing so we find that the school still appears to be performing less well than the national trend. One other factor in the virtual comparator analysis is stage of leaving. To test if this is what is causing the difference we can deploy the filter for Stage to ascertain the impact of this factor on attainment. By keeping the ASN filter to Other and selecting pupils who left from S4, we begin to see something interesting. This time, attainment is generally in line with the national trend: By isolating and examining data on S4 leavers in this way, it can be concluded that a high number of the school s high-attaining pupils appear to be leaving at the end of S4. 14
Let s turn our attention to the school s students remaining until the end of S6: Discussion: What can we learn from each stage of leaving graph for this measure and, collectively, what do they tell us about the attainment of the school s leavers as a whole? What positives can be taken from the data on the attainment of the school s S6 leavers? How can the data help inform the school s planning for improvement? Does it help identify a priority area? If so, what is it? Our conclusion so far is that pupils seem to be leaving the school with lower attainment than average given their background and that one reason for this is because some are leaving earlier than in other schools. However, the increasing post-school participation national measure does show that the positive destination rate is still high. It may be that these pupils are going on to employment, but possibly to jobs that require fewer qualifications than those which could be achieved through further study at school. Consequently, the school may wish to address its staying on rate as doing so could increase the attainment of those pupils in S4 who are currently leaving with fewer qualifications than they have the potential to attain. Of course, a school s socio-economic context can have a powerful influence on such factors and it may be that this school is in an area of high employment where pupils are attracted by jobs that are more highly-paid relative to those available to their peers elsewhere in the country. To test this hypothesis we can look at the attainment by SIMD for all leavers for the local authority: 15
In doing so, we find that in many SIMD bands, the authority as a whole performs better than the school and is generally in line with the national average. This would suggest that there are schools in the same local authority with which this school could enter into professional dialogue in order to explore further how best to encourage pupils to continue their studies and attain more highly by the time they leave S6. The school may also wish to analyse the difference in attainment between S4/S5 leavers and those leaving school at the end of S6. A review of strategies to raise the attainment of pupils in specific SIMD deciles may also render a positive impact. 3. Exploring a curricular area at school level As well as the four national measures on the Insight dashboard, a number of additional local measures enable schools and local authorities to take a closer look at performance data relevant to their own curriculum, subjects and courses. This range of local measures includes: Attainment in Selected d Course Percentage of Resulted Entries Attainment in Selected Ungraded Course Curricular Areas all curricular areas and single curricular area Before proceeding, let s briefly consider each of these: Attainment in Selected d Course Percentage of Resulted Entries The term d Course refers to SCQF level 5 or above in the new national awards (and Intermediate 1 in pre-existing qualifications), while Resulted Entries means those pupils who were actually presented for a course award. 16
With this measure, users can see how pupils performed in a particular subject or course at a particular level. By deploying the available filters it is also possible to drill down into the data to identify trends in performance. Attainment in Selected Ungraded Course This is similar to the measure above, except that it looks at ungraded courses (e.g. National 4). Consequently, the tables of the measure reflect the main difference that ungraded courses are marked pass or fail. Curricular Areas These measures allow users to analyse how their learners performed in the different curricular areas, thereby providing understanding of the balance of a school s curriculum and the comparative strengths of those curricular areas. The phrase base centre cohort refers to the number of pupils there are in total for the filter being considered. For example, if S4/S5/S6 is selected in the Stage filter, then the base centre cohort is the total number of pupils in the school in S4/S5/S6. If only S4 is selected, then the base centre cohort is the number of S4 pupils in a school. For the purposes of Insight, there are ten curricular areas in the tool: Wider Achievement English and Gaidhlig Expressive Arts Health and Wellbeing Languages apart from English Mathematics Religious and Moral Education Sciences Social Subjects Technologies The inclusion of wider achievement provides the opportunity for users to consider both the breadth and depth of learners accomplishments. In keeping with a key aim of Curriculum for Excellence, additional wider achievement awards will be incorporated into Insight over time. To do so, these awards must be in line with Curriculum for Excellence technical requirements such as being able to handle individual pupil data and be aligned to the Scottish Credit and Qualifications Framework. Schools can use the local measures to consider the performance of learners in each curricular area. An initial starting point might be to compare the percentage of total resulted entries in one curricular area against the curricular area s benchmark data. This can be done by: Selecting the curricular areas single curricular area local measure 17
Under the resulting graph: Choose the appropriate curricular area from the Curricular Area filter below the graph before clicking Update. For example, the graph below shows that, over a three year period, this school has consistently had around 10% more resulted entries in the curricular area of social subjects (orange) than its virtual comparator (grey), local authority (blue) and national benchmarks (red). An example of what the graph tells us is that in 2014 approximately 27% of all curriculum entries in S4/5/6 were in social subjects: In addition, analysis of pupils with resulted entries as percentage of base centre cohort for social subjects also reveals a similar differential: 18
By deploying the appropriate filter this data can also be broken down according to stage. For example, selecting S4 in the Stage filter reveals that (although the margin of difference is smaller) the school s resulted entries as a percentage of the base centre cohort in social subjects is high and well above the benchmarks: 19
While taking S5 and S6 together reveals the following: Again, a clear positive difference is apparent between S5/S6 pupils with resulted entries as percentage of base centre cohort and comparative data. The further use of various filters allows users to break down this information even more by isolating particular groups of learners for analysis. A next step in considering performance in a curricular area could be through the use of attainment in selected graded course percentage of resulted entries, available from the local measures menu. This allows a school to consider learners attainment in courses in each subject of a given curricular area. For instance, below is a breakdown of results in graphic form for Higher Geography in the same school over a three year period. As with other measures, results are compared against local authority and national data. 20
Higher Geography: One of the first things that can be noted from the graph is the percentage of A grades in Higher Geography attained by pupils over the three years and the fact that these are notably higher than both the local authority and national performance. Graphic representations are also available for history and modern studies courses: Higher History: 21
Higher Modern Studies: To assist users in their analysis, the data is also available in tabular form. Let s use the Aggregated Tabular Data to compare the performance of the school s learners across the three social subjects Higher courses over a three year period: Higher Geography: Establishment Year # A % A # A to B % A to B # A to C % A to C # A to D % A to D School 2012 28 50.91 40 72.73 44 80 50 90.91 Local authority 2012 78 41.27 125 66.14 155 82.01 169 89.42 National 2012 2331 34.66 4000 59.48 5411 80.46 5901 87.75 School 2013 24 45.28 33 62.26 42 79.25 46 86.79 Local authority 2013 81 36.65 129 58.37 173 78.28 188 85.07 National 2013 2245 33.94 3765 56.92 5218 78.88 5677 85.82 22
School 2014 24 46.15 34 65.38 45 86.54 49 94.23 Local authority 2014 80 37.38 124 57.94 177 82.71 187 87.38 National 2014 2294 28.6 4148 51.72 5838 72.79 6507 81.13 Higher History: Establishment Year # A % A # A to B % A to B # A to C % A to C # A to D % A to D School 2012 15 46.88 28 87.5 30 93.75 31 96.88 Local Authority 2012 45 35.16 78 60.94 103 80.47 117 91.41 National 2012 2436 30.31 4889 60.84 6683 83.16 7073 88.02 School 2013 24 60 34 85 39 97.5 39 97.5 Local Authority 2013 45 32.61 87 63.04 120 86.96 128 92.75 National 2013 2933 33.22 5728 64.88 7804 88.4 8256 93.52 School 2014 12 28.57 24 57.14 36 85.71 37 88.1 Local Authority 2014 28 16.18 75 43.35 127 73.41 145 83.82 National 2014 3273 31.58 6518 62.9 8918 86.06 9500 91.67 23
Higher Modern Studies: Establishment Year # A % A # A to B % A to B # A to C % A to C # A to D % A to D School 2012 18 37.5 33 68.75 44 91.67 46 95.83 Local authority 2012 44 24.31 104 57.46 164 90.61 171 94.48 National 2012 2153 30.97 4207 60.51 5904 84.91 6321 90.91 School 2013 21 42.86 31 63.27 41 83.67 45 91.84 Local authority 2013 68 36.56 103 55.38 151 81.18 165 88.71 National 2013 2062 30.22 4103 60.13 5824 85.35 6195 90.78 School 2014 13 26 30 60 38 76 42 84 Local authority 2014 42 19.18 101 46.12 148 67.58 172 78.54 National 2014 2264 26.54 4576 53.65 6755 79.2 7391 86.66 Discussion: In each band for these three years (i.e. from % A to % A to D) how does performance in: Higher Geography Higher History Higher Modern Studies compare against local authority and national comparisons? 24
In each band for these three years (i.e. from % A to % A to D) how does performance in: Higher Geography Higher History Higher Modern Studies compare against performance in the other two subjects? By taking account of your analysis of the graphs for the three Higher courses and your conclusions from the above questions: What follow-up questions would you recommend the social subjects faculty, along with its senior managers, should ask in order to build on the performance of its learners in each of these courses? Considering the data offered by all the tool s measures will encourage schools and other stakeholders to develop further their awareness of what lies behind the information presented. This can be achieved by using both the national and local measures, such as those illustrated here, as the basis of professional dialogue around the performance of learners, not only across the school as a whole but in each curricular area. By engaging in professional dialogue based on an analysis of Insight data, schools will be able to plan for improvement and develop points for action aimed at having a specific impact on learner achievement. 25