U N I V E R S I T Y O F B E R G E N What Does Research Say about Adaptive Learning? Gyldendal Seminar 19 April 2016 Professor Barbara Wasson
SLATE What is Adaptive Learning? Digital learning systems are considered adaptive when they can dynamically change to better suit the learning in response to information collected during the course of learning rather than on the basis of preexisting information such as a learner s gender, age, or achievement test score. Adaptive learning systems use information gained as the learner works with them to vary such features as the way a concept is represented, its difficulty, the sequencing of problems or tasks, and the nature of hints and feedback provided. US Department of Education, Office of Educational Technology (2013) 2
Adaptive Learning: Motivation Practical challenges in teaching and learning: content that is too easy or too hard tends to de-motivate students, boring them if it s too easy and frustrating them if it s too hard; students come to a class with fundamentally different levels of prior knowledge; costs of education prevent a student from receiving the oneon-one instructor attention that has been shown to make a major improvement in learning." Oxman, S., & Wong, W. (2014). White paper: Adaptive learning systems. Integrated Education Solutions, p. 6-7. 3
Adaptive Learning: Motivation Adaptive learning potential: reduce coursing drop-out rates; being more effective at achieving outcomes; being more efficient for students, helping them achieve outcomes faster; freeing up teacher / faculty to focus on direct assistance where it is needed most. " Oxman, S., & Wong, W. (2014). White paper: Adaptive learning systems. Integrated Education Solutions, p. 6-7. 4
TEL: Historical Overview? 5
TEL: Historical Overview 1960 1970 1980 1990 2000 2010 6
TEL: Historical Overview CAI 1960 1970 1980 1990 2000 2010 CBL 7
TEL: Historical Overview CAI CE ITS 1960 1970 1980 1990 2000 2010 CBL microworlds 8
ITS: Motivation Compelling evidence that individualised tutoring engenders the most effective and efficient learning across an array of domains (Bloom, 1994, Woolf, 1998) Time on task Appropriate feedback 9
Domain Knowledge Component System Learning Component Student Knowledge Component Instructional Planning Component System Control (Blackboard) Communication Component 10 rule-based cognitive science informed diagnose errors and tailor remediation
ITS: Early results SHERLOCK (Lajoie & Lesgold, 1989, 1991) used to train Air Force technicians to diagnose problems in the electrical systems of F-15 jets creates faulty schematic diagrams of systems for the trainee to locate and diagnose provides diagnostic readings allowing the trainee to decide whether the fault lies in the circuit being tested or if it lies elsewhere in the system feedback and guidance are provided & help is available if requested! Trainees who spent 20-25 hours on Sherlock were as proficient as technicians who had been on the job 4 years or longer 11
ITS: Early results Cogntitive Tutors (e.g., Anderson et al, 1985, 1995; Koedinger et al. 1997)) CMU research group cognitive tutors used on a regular basis in many schools in USA (algebra, geometry) Pane et al. (2014) studied effect in middle (8700 students) and high schools (16 800 students) over 7 states! Analysis of posttest outcomes on an algebra proficiency exam finds no effects in the first year of implementation, but finds evidence in support of positive effects in the second year. The estimated effect is statistically significant for high schools but not for middle schools; in both cases, the magnitude is sufficient to improve the median student s performance by approximately eight percentile points. 12
TEL: Historical Overview CAI CE ITS CSCL Mobile Learning 1960 1970 1980 1990 2000 2010 MOOCs ALEs Maker kits CBL microworlds CSILE Telelearning Environments Participatory Environments 13
Example-tracing tutors built without programming CTAT (cognitive tutor authoring system) generalised examples of problem-solving behaviour 4-8 times more cost-effective to develop Support a variety of pedagogical approaches including step-based problem solving, collaborative learning, educational games, and guided invention activities Can be embedded in LMS (e.g., Tutorshop) or MOOCs (e.g., edx) Focus on tutoring at scale (inner loop tutor moved from server to client) 14
15
16
17
TEL: Historical Overview CAI CE ITS CSCL Mobile Learning trialogical learning 1960 1970 1980 1990 2000 2010 MOOCs ALEs Maker kits CBL microworlds CSILE Telelearning Environments Participatory Environments 18
Challenge: Recognise and Explain What and Why wrong answer! New Problem / Task vs Feedback & Feedforward (incorporate what we know about AfL) 19
Challenge: Empirical Evidence Model-tracing & Example-tracing tutors vs Algorithmic adaptive systems 20
Challenge: Dashboard design & content Computer Science Perspective vs Teacher Perspectives (what do teachers find relevant, and for what purposes) (Verbert et al., 2014) 21
Challenge: Real-time data...devise ways for real-time data to provide teachers with insights to guide their instruction (Siemens & Gasevic, 2012)... to understand how real-time data can support teachers in monitoring their students work, and in deciding on the most appropriate instructional interventions (Matuk & Linn, 2016) 22
Challenge: Data Literacy and Use Digital competence vs Digital competence with Data literacy and use (Wasson & Hansen, 2016) 23
Challenge: Use of Teacher s Tacit Knowledge 24 System Alone vs System + Teacher...opportunity to enhance the usefulness of real-time data by allowing teachers to integrate their own insights (Matuk & Linn, 2016)
Challenge: Evaluation difficulties Careful evaluation should be undertaken to ensure that these complex systems do what they claim. Testing and field trials during the design and early development of the system (prototypes, short time frames of use) Evaluation, in situ, after completion of the system to support formal claims learning and learning outcomes (completed system, longer time frames of use, for example, in classrooms) 25
Access to adaptive learning systems Technology Rich Learning Environments Authoring Tools Gradual Acceptance Educational institutions New pedagogical methods Political 26
Conclusions Great potential Movement from the lab to the classroom / online Evidence for model-tracing and example-tracing approaches Issues of scalability are being addressed, including authoring Hype danger Challenges Evidence of learning and efficacy Incorporate assessment for learning results Empirical evidence of the usefulness of dashboards (from technical aspects to teaching and learning aspects) 27
References Alven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Ringenberg, M. & Koedinger, K.R. (2016). Example-Tracing Tutors: Intelligent Tutor Development for Nonprogrammers. International Journall of Artificial Intellignce, 26, 224-269. DOI 10.1007/ s40593-015-0088-2 Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned.the Journal of the Learning Sciences, 4(2), 167 207. Koedinger, K. R., Anderson, J. R., Hadley,W. H., &Mark,M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30 43. Matuk, C. & Linn, M.C. (2016). Teachers reflections on the uses of real-time data in their instruction. AERA. Oxman, S., Wong, W. (2014). White paper: Adaptive learning systems. In: Integrated Education Solutions. Pane, J. F., Griffin, B. A.,McCaffrey, D. F., & Karam, R. (2013). Effectiveness of Cognitive Tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 0162373713507480. doi: 10.3102/0162373713507480. Rau, M. A., Aleven, V., Rummel, N., & Pardos, Z. (2014). How should intelligent tutoring systems sequence multiple graphical representations of fractions? A multi-methods study. International Journal of Artificial Intelligence in Education, 24(1), 125 161. doi:10.1007/s40593-013-0011-7. Rau, M. A., Aleven, V., & Rummel, N. (2015a). Successful learning with multiple graphical representations and self-explanation prompts. Journal of Educational Psychology, 107(1), 30 46. doi:10.1037/a0037211. Rau, M. A., Michaelis, J. E., & Fay, N. (2015b). Connection making between multiple graphical representations: A multi-methods approach for domain-specific grounding of an intelligent tutoring system for chemistry. Computers & Education, 82, 460 485. doi:10.1016/j.compedu.2014.12.009. Wasson, B. & Hansen, C. (2016). Data literacy and use for teaching. In P. Reimann, S. Bull, R. Lukin, B. Wasson (Eds.) Measuring and visualising competence development in the information-rich classroom, 56-74. New York: Routledge. 28
slate.uib.no #slateresearch FB: Centre for the Science of Learning and Technology