International workshop Fostering Innovation and Improvement in Education: the Contribution of Longitudinal Information Systems WORKSHOP THEMES & RESULTS FROM THE OECD/CERI SURVEY Stéphan Vincent-Lancrin Carlos González-Sancho OECD Directorate for Education and Skills New York City, 30 June 1 July 2014
Outline Welcome and thanks CERI Innovation Strategy for Education and Training Why this work and this workshop The CERI survey on longitudinal information systems Results Conclusions and steps forward
innovation in education
Innovation in education Technology Research and Development Innovation in education School organisation System organisation
General Purpose Technology Technology Longitudinal information systems in education Research and Development Innovation in education School organisation Next generation = integration of statistical data systems and learning management systems with quick feedback and visualisation tools (expert systems) System organisation Building on OECD/SSRC/Stupski workshop, October 2010 Engage in the discussion about Big Data in education
why this work and this workshop
The opportunities Quick feedback to stakeholders: A new tool for formative assessment and the design of quick remedial strategies (an expert system for teachers?) A tool for cultural change (personalisation?) Platforms to network and mobilise practical knowledge: connect teachers and schools with same concerns (learning communities) Platforms to post relevant instructional material to support teachers and improve knowledge management and develop a stronger educational industry Platform to improve efficiency and reduce administrative costs (admission process, student transfers, statistical collections, etc.) Creation of a better data infrastructure for educational research and the evaluation of educational innovations
Several tricky questions Big Brother and Gattaca: Privacy and safety: who should have access to what? What level of details? For how long? For what purposes? Quality and speed of feedback: how to ensure the data are of good quality (accurate, comparable)? That they are relevant? Accountability: is it the way to make the systems relevant to people? is there a risk of rejection because of a use for accountability purposes (sanctions/rewards)? Are evaluators (inspectors, etc.) still needed? Data driven education: an inappropriate narrowing down of educational objectives? How to ensure people use the data that are collected for improvement and innovation?
Objectives of the workshop Get an idea of the state of play of current systems and identify the new horizons for next-generation systems Identify the most important policy questions and define key recommendations that would allow one to reap the benefits of these systems Identify what role international collaboration and exchange could play in this regard (Learn and continue the international conversation)
The CERI survey on longitudinal information systems
64 systems from 32 countries 26 systems surveyed in 2010 20 systems surveyed in 2013 outside the US 18 US state-wide systems from 2013 DQC survey Countries/economies: Australia [3], Austria [2], Belgium [2], Brazil [2], Canada [2], Chile [2], Czech Republic, Estonia, France, Germany, Hungary, India, Israel, Italy, Japan, Korea [2], Lithuania, Mexico, Netherlands [3], New Zealand, Norway [2], Portugal, Slovak Republic [2], Slovenia, South Africa, Spain [2], Sweden [2], Turkey, UK [2], US [20] Survey administered to systems managers in operating agencies US state-wide systems: equivalent survey items identified in consultation with DQC
Survey sections 1. Goals of the system 2. Data model 3. Coverage and frequency of collection 4. Data linkages 5. Quality processes 6. Access and privacy 7. Comparison possibilities 8. Accountability usage 9. Instructional support, networking facilities and PD 10. Other features Good but not perfect overlap between OECD and DQC surveys: results presented separately Results are preliminary updates and more quality checks pending
Most systems are recent, public and cover K-12 schooling at national/state level About 80% in use <10 years, of which 10% <5 years About 20% in place 10+ years, of which 5% set up 20+ years ago Systems cover compulsory stages of schooling, but not only About 60% cover also early (i.e. pre-primary) education Almost 50% cover either post-secondary or tertiary education Less than 10% are HE-systems only and up to a third provide comprehensive P-20 coverage Mainly found in the US but also in Estonia, Belgium, Slovakia, Lithuania and the Netherlands Most are public and national or regional/state level, with data on all or a representative sample of the students in the jurisdiction
Longitudinal identifiers and linkages 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% School ID Student ID Teacher ID Course code All 4 Student-Teacher link OECD 2010-13 DQC 2013 Schools and students are uniquely identified by most systems Fewer systems, especially outside the US, provide teacher and course identifiers Systems tend to enable linkages between most of the data elements available for identified entities
Student assessment data 100% 90% 80% Course Grades Skills in thinking and creativity Formative assessment data Graduation Item/exercise/question level data Still largely focused on conventional attainment and summative performance indicators 70% 60% 50% 40% 30% 20% 10% Data on students soft and generic skills rarely available Formative assessment and item/question level data missing from most systems 0% OECD 2010-13 DQC 2013
Teacher data Years of Service Level of education 100% 90% Subjects taught Evaluation Professional development Information systems with teacher-level data are much more common in the US 80% 70% 60% Teacher data mainly focus on credentials, seniority and teaching duties 50% 40% 30% 20% 10% Data on teacher evaluation and professional development are absent from most systems 0% OECD 2010-13 DQC 2013
School data 100% 90% 80% Administrative Networks and programmes Type of school Evaluation Virtually all systems provide school admin data and are able to group schools by type/status 70% 60% 50% 40% 30% 20% 10% School evaluation data is rarely found outside the US Systems could make progress in recording school participation in networks and innovation programmes 0% OECD 2010-13 DQC 2013
Data access restrictions 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Full access to anonymised student-level data Administrators Principals Teachers Parents Researchers OECD 2010-13 DQC 2013 Full access to data remains the privilege of education authorities and school leaders, even after anonymisation Re-identification risks may persist, but limitations of access are at odds with effective data-use Access to elements such as assessment results is highly restricted
Speed of feedback 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Real Time <1 month >1 month Administrators Principals Teachers Parents Students Researchers OECD 2010-13 only Timeliness of feedback is a critical condition to maintain data value Most systems outside the US take more than 1 month to make data available, regardless of access rights many impose >6 months delays Cited reasons for delay include data cleaning and anonymisation
Analysis and comparison tools 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Integrated tools Between schools Only about half of the Between students Between teachers Peer comparison systems integrate tools for comparison OECD 2010-13 DQC 2013 Dashboards, automated reports or other tools for data mining and analysis When such tools are available, they more often enable comparisons between schools than between individuals Tools tend to allow peer-comparisons
Quality assurance mechanisms 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Support for data reporting Training for data reporting Automated rules Check on subsamples OECD 2010-13 DQC 2013 n/a Support and training for data reporting available on many systems Both definitions and technical standards Automatisation of quality checks and inconsistency alerts missing in about a third of the systems Key when data are directly imported from multiple subsystems
Use of systems for high-stakes accountability and ranking 100% 80% 60% 40% 20% 0% 100% 80% 60% 40% 20% 0% System data used for high-stake assessment of Schools Students Teachers OECD 2010-13 DQC 2013 System data used to grade/rank performance of Schools Teachers OECD 2010-13 DQC 2013 Systems often used to support high-stakes accountability, especially in the US e.g. admission decisions, career promotion, financial sanctions or school closure Systems less often used to inform publicised performance evaluations, especially outside the US May contribute to negative perception as punishing tools
Training and materials to improve instruction 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Training to use the data to inform instruction Communities of practice Teacher access to materials Students/parents access to materials n/a OECD 2010-13 DQC 2013 Some systems, especially in the US, provide training to use data to inform instruction However, very few are linked to repositories of digital materials that may be used during or to complement classroom instruction Data systems still far from becoming expert systems with recommendation engines and pedagogical tagging of materials
Conclusions
Summary of survey results Student and/or school data systems mainly. Teacher IDs and course codes needed in order to effective support instruction Richer student assessment data needed. This may involve more efficient methods of data collection and new instruments Data access restrictions remain pervasive, marked differences by role Current speed of feedback at odds with support to ongoing interventions Analysis and comparison tools remain insufficiently integrated Automated inconsistency rules but very limited ability of stakeholders to check the data (expertise) High-stake uses are common and stated as goals. Other uses need to be promoted to change perceptions about aims of data systems Still far from becoming expert systems with the ability to provide personalised advice and support for teaching and learning
OECD / CERI Thank you stephan.vincent-lancrin@oecd.org carlos.gonzalez-sancho@oecd.org http://www.oecd.org/edu/innovation