Centre for Evaluation & Monitoring SOSCA Feedback Information
Contents Contents About SOSCA... 3 SOSCA Feedback... 3 1. Assessment Feedback... 4 2. Predictions and Chances Graph Software... 7 3. Value Added Software... 9 page 2
About SOSCA SOSCA is a quality, computer delivered, curriculum-based monitoring system that allows teachers to assess pupils performance in Maths, Science and Reading at the end of year 9/S2 (age 13+). It is a robust, accurate, confidential and timely information system that equips teachers and school managers with powerful information about their school and is aimed at helping them to improve school performance. SOSCA is part of a suite of monitoring systems produced by one of the UK s leading educational research centres CEM, based at Durham University. CEM is recognised around the world for its pioneering educational research and school monitoring systems and is a trusted supplier of education assessment systems. CEM delivers over 1 million tests to pupils every year, and works with schools in over 36 nations. SOSCA Feedback The Information you receive falls into three main categories: 1. Assessment Feedback SOSCA Standardised Scores. Band Profile Graphs. Individual Pupil Records. About SOSCA 2. Predictions and Chances Graph Software Predictions to GCSE, Scottish Qualifications and 5-14 attainment Levels (as appropriate) based on each pupil s SOSCA Test scores. Pupil Chances Graphs, based on an individual s test score, giving a guide to likely attainment at GCSE, Scottish Qualifications and 5-14 attainment Levels. Subject-level Chances Graphs. 3. Value Added Software Value Added Feedback from PIPS P7 and MIDYIS to SOSCA (where data are available). Value Added Feedback from SOSCA to GCSE / Scottish Qualifications. Accessing and importing the Feedback All feedback is available to download from the secure SOSCA+ website. Feedback is presented as Microsoft Excel workbooks and should be easy to import into your MIS system. page 3
Assessment Feedback 1. Assessment Feedback i. SOSCA Standardised Scores What these mean SOSCA assessment results are standardised against a sample of schools that took part in the tests for that curriculum. For each pupil there are standardised scores for each assessment: Maths, Science and English Reading (plus their subtests as shown in the diagram below). The results are standardised to have a mean of 100 and a standard deviation of 15. Why are they useful? These scores allow you to judge how well your pupils have performed on the SOSCA assessments, which are curriculum based tests for the end of Year 9/S2, in relation to other pupils and also how they have performed relative to the sample average. Information is available at the pupil, class and cohort level. Subject Feedback Example The reported subjects depend upon the curriculum followed, so for example the feedback for Scotland would look like the table below. page 4 ii. SOSCA Grade Bands How these are calculated The pupils' scores for each assessment are given a SOSCA Band A to D (where A is the highest and D the lowest). The bands are constructed using quartiles each band containing 25% (a quarter) of the pupils in the SOSCA sample. A pupil scoring in band A has scored in the top 25% of the sample.
Band profile graphs are provided for each assessment Maths, Science and English Reading, and also each sub-assessment. A school with a profile like the standardisation sample, in terms of average attainment and range of performance, would have 25% of pupils in each band (this is shown by the red broken line on the graph below). What these graphs tell us. The graph shows the proportion of pupils falling into each of the four performance bands A D and allows schools to see how their pupils performed compared to the entire SOSCA sample of pupils. This provides an overview of the attainments of their own pupils. Assessment Feedback Why are they useful? Monitoring your pupils' attainment over a number of years helps you to anticipate future expected performance in external examinations and also gives a measure of the range and level of achievement of the pupils you are working with. iii. Individual Pupil Records (IPRs) Individual Pupil Records display, for each pupil, a personalised sheet summarising all the SOSCA attainment information for that pupil in addition to the name, date of birth, sex and student number of each pupil. page 5
Assessment Feedback What the IPR Tells us (for each assessment) The SOSCA band (A to D, with A being the highest) into which the pupil s performance falls. The stanine (one of nine divisions, with 9 being highest) into which the pupil s performance falls. The percentile (the percentage of the sample that this pupil performed better than) corresponding to the pupil s performance. The pupil s standardised score. The pupil s standardised score in each subject, together with 95% confidence bands. (Confidence bands represents the range of probable scores for that pupil if the test were to be repeated many times. There is only a 5% chance the score would fall outside this range.) page 6 Why is this useful? The graphical representation of a pupil s performance on the SOSCA Assessments allows you to quickly and easily identify any areas for concern. Where the confidence bands do not overlap between assessments or sub assessments this indicates the pupil has performed significantly better or worse in that area compared to other areas. A significantly low score on a particular assessment may mean that intervention is needed in order that the pupil achieves his or her full potential.
2. Predictions and Chances Graph Software What this includes: Predictions and Chances Graphs for GCSE grades (Full, Short and Vocational), Scottish Qualifications and 5-14 Attainment Levels, as applicable. What this shows: i. Pupil level predictions worksheets These worksheets show point predictions for each pupil in GCSE, Scottish Qualifications or 5-14 attainment levels for each subject, as applicable. Change Points to Grades With the 'Points To Grades' button the user can choose to display the predictions as points, for example as 4.3, or as letters, e.g. D/E. Why is this useful? Predicted grades can be used to aid the target setting process and can be analysed at the pupil, form, subject and cohort level. In addition to the chances graphs (described later) they can be used to monitor pupil performance against the average progress made by pupils of similar ability and used to help inform option choices. Predictions & Chances Graph page 7
Predictions & Chances Graph page 8 ii. Pupil level chances graphs What these graphs show These example graphs show the pupil's probability of achieving various levels/grades in different subjects. Chances graphs are available for GCSE subjects (full, short and vocational), Standard Grade subjects and 5-14 levels. Examples from 5-14 and GCSE are shown below. Summary Chances Graph Sheet showing 5-14 levels Pupil level chances graph showing GCSE Maths Why is this useful? Chances Graphs are an excellent way of working with pupils to show what range of grades they are likely to achieve in external examinations. They can be used to: Warn more able pupils against complacency. Encourage less able pupil to work hard and stay focussed. Show that a predicted grade is not a foregone conclusion. Pupils of a given achievement level ultimately achieve a range of grades. Help parents understand the realistic likely achievements of their child in external examinations. Illustrate different chances of success in different subjects. Aid option choices. iii. Subject Level Chances Graph Summary graphs at the subject level inform likely overall performance in each subject.
3. Value Added Software What is Value added? Value Added measures the amount of progress made by pupils over a certain period of time, compared against pupils of similar abilities in other schools. Why is this useful to know? If one school is increasing the achievement level of its pupils more than other schools, then its pupils gain an additional advantage. This relative advantage is what is called Value Added. SOSCA 3 Levels of Value Added SOSCA has three levels of Value Added available. 1. Pupils who have taken MIDYIS in Year 7 or 8/S1 can usually be matched to pupils taking SOSCA. 2. Pupils who have taken the PIPS in P7 can usually be matched to pupils taking SOSCA (as long as we know their UPN and their primary school's number or code) and primary schools/eas agree that data can be shared between schools for this purpose. 3. GCSE/Scottish Examination GCSE / Scottish Qualifications SOSCA MidYIS PIPS (P7) results for pupils who have taken SOSCA provide the outcome for the third level of Value Added. Predictions Value Added Value Added Software Feedback information Although the baseline and outcome measures differ between the three, the style of Value Added Feedback is similar. The examples shown here may be drawn from any of the three. i. Value Added Feedback: Summary Worksheet What this worksheet shows This sheet provides Value Added information for each pupil in: the three SOSCA subjects (Maths, Science and Reading) from the MIDYIS or PIPS baselines page 9
Value Added Software GCSE / Scottish Examination subjects from the SOSCA assessments Example of the Value Added feedback from MIDYIS to SOSCA. Feedback for the PIPS to SOSCA stage is very similar. That for SOSCA to GCSE or Scottish Qualifications covers many more subjects: ii. Subject Reports for each subject in SOSCA or at GCSE/ Scottish Qualifications For each subject, this report (below) allows the user to display a scatter graph and/or a Pupil Level Residuals (PLR) table. Here we refer to the example for SOSCA to Standard Grade, but similar feedback is provided for GCSE and for the SOSCA Assessments from the MIDYIS or PIPS baselines. page 10
Example of Scatter Graph and Pupil Level Residuals Table What the Scatter Graph Shows Each Scatter Graph shows the regression line for that subject with your pupils results superimposed (SOSCA test scores versus actual GCSE/ Scottish Qualifications results). A regression line describes the relationship between the SOSCA score and each external examination. The residuals table shown below the graph details for each pupil: Their overall SOSCA test score. Their expected performance predicted using the SOSCA test scores. Their achieved result. Their raw residual or value-added, i.e. the difference between the achieved result and the expected result. Their standardised residual this is the raw residual re-scaled to show its position in the whole sample. Why is this information useful? These scatter graphs provide an overview of pupils Value Added results. Pupils plotted above the line have performed better in that particular subject than expected given their SOSCA test scores they have made more progress relative to other pupils of similar ability, therefore achieving positive Value Added. page 11 Value Added Software
Value Added Software Pupils who fall on the line have performed exactly as we would have expected given their SOSCA test scores; these pupils achieved zero Value Added. Those pupils below the line have negative Value Added and have made less progress than those of similar ability. iii. Value Added Summary Graph This graph is available for: SOSCA Assessments. GCSE (full, short and vocational courses). Scottish Qualifications. It shows the average raw residual (difference between predicted and actual performance) for each subject analysed. It shows how well pupils in each subject performed in comparison with similar pupils in the same subject in other schools. Graphs for PIPS to SOSCA or MIDYIS to SOSCA show the SOSCA subjects. page 12...Why is it useful? A positive score on the graph means that, on average, pupils in your school did better in a subject than pupils of a similar ability in the rest of the SOSCA sample. By the same token, a negative score means that, on average, your pupils did not do as well in that particular subject when compared to pupils of similar ability who took the subject.
iv. Average Standardised Residual Graph What are Standardised Residuals? Standardised Residuals are raw residuals or value added rescaled so that all measurements are presented on the same, fair scale. They allow comparisons to be made fairly between subjects and from year to year. This graph, available for: SOSCA Assessments. GCSE (full, short and vocational courses). Scottish Qualifications. shows the average standardised residual for each GCSE / Scottish Qualification subject. Examples of Average Standardised Residuals Graphs: Value Added: MIDYIS to SOSCA and SOSCA to Standard Grade. Value Added Software why is this graph useful? Average Standardised Residual graphs allow you to put your subjects performance into context and note, over time, whether any trends (good or bad) are occurring in your pupils achievements. Subjects performing as expected lie completely within the confidence bounds. Subjects may be positive one year, page 13
negative the next and are displayed in white. These changes in results are not significant and can be attributed to the chance variation in the different groups of pupils taught each year. Here achievement is roughly inline with predictions. Subjects performing in excess of expectations are positive, displayed in blue and lie outside the inner confidence bound. Here the performance of pupils in a subject is largely beyond statistical error and pupils are achieving, on average beyond their expectations. Subjects performing below expectations are negative, displayed in yellow and lie beyond the inner confidence bound. Here the performance of pupils in a subject is largely beyond statistical error and pupils are achieving, on average below their expectations. Confidence Bounds The two shades of grey behind the bars represent confidence bounds. These confidence bounds allow you to see whether a positive or negative standardised residual is significantly different from zero or whether it can be attributed to chance variation. page 14
Notes page 15
Centre for Evaluation & Monitoring For more information visit: www.soscaproject.org Email: sosca@cem.dur.ac.uk Tel: +44 0191 334 4210 www.cemcentre.org