Process data as a window into students minds?

Process data as a window into students minds? The exploita7on of computer- generated log files University of Luxembourg

Overarching goal: Explore the poten-al of computer- generated log files with the PISA 2012 problem solving data 2

Some ques-ons in the back of our head... Do log files allow us 1) looking into students minds? 2) moving from the What (students can do) to the How (they actually do it)? Is there a hidden poten7al. 1) for repor7ng in large- scale assessments? 2)...for research? 3) for educa7onal policy? 3

Log files: The solu7on to everything? Gartner s hype cycle for emerging technologies Where do we stand?

Agenda What was Problem Solving in PISA 2012? What did we do? The ra7onale Process data in PISA 2012 What did we learn? The results What s ahead? Implica7ons

PISA 2012 Problem Solving in brief Innova0ve PISA domain in 2012 85 000 students in 44 countries/economies took an interna7onally agreed 40- minutes test - on computers - to assess students capacity to engage in cogni7ve processing to understand and resolve interac0ve and dynamically changing problem situa0ons where a method of solu7on is not immediately obvious - in addi7on to the 2h paper- based test (mathema7cs, reading, science) Copyright by F. Avvisa7 6

PS and the other domains Mean R 2 of other domains: 77% overlap Mean R 2 of Complex Problem Solving with other domains: 61% overlap Copyright OECD (2014)

Mean score 570 560 550 540 530 520 510 500 490 480 470 460 450 440 Korea Hong Kong-China Chinese Taipei Finland Estonia Germany Czech Republic United States Norway Denmark Portugal Russian Fed. Strong performance in problem solving Poland Slovenia Israel Singapore Japan Macao-China Shanghai-China Canada Australia England (U.K.) France Italy Netherlands Austria Belgium Ireland Sweden Slovak Republic Spain Serbia Croatia Hungary Turkey Chile Average performance of 15-year-olds in problem solving Fig V.2.3 430 420 410 400 Montenegro Bulgaria Brazil Malaysia U.A.E Uruguay Colombia 390 Low performance in problem solving Copyright by F. Avvisati

Problem Solving Sample Question 2 Climate Control This is a harder item Level 4 on You have no instructions the problem- solving for your new scale air conditioner. You need to work out how to use it. Find whether each control influences temperature and Students must engage with the humidity by changing the sliders. machine, and use the feedback and Draw lines in the diagram on the right to show informa7on what each uncovered control to reach a influences. solu7on: it is an interac0ve problem This main demand is represen7ng and formula7ng (knowledge acquisi0on)

PISA Crea7ve Problem Solving Tasks

Possible insights from this item: How much 0me did students spend to solve this item? How many «experiments» did they set up (varying one thing at a 7me)? Did they press all the controls «at random», or according to a systema0c plan? Did some students try to guess the right answer, without gaining first all the relevant informa7on? Copyright by F. Avvisa7 11

The aim of this TJA fellowship was to showcase log file analyses on the basis of PISA 2012 data. Because of its dynamic nature and the interac@on between problem and student, the problem solving assessment is a par-cular good candidate for these analyses. All analyses on the basis of the task Climate Control. 12

Agenda What was Problem Solving in PISA 2012? What did we do? The ra7onale Process data in PISA 2012 What did we learn? The results What s ahead? Implica7ons 13

14 PISA Creative Problem Solving Tasks

15

Top Control Temperature Central Control Humidity Bottom Control 16

Axis 2: Assessment instruments VI

Agenda What was Problem Solving in PISA 2012? What did we do? The ra7onale Process data in PISA 2012 What did we learn? The results What s ahead? Implica7ons 18

How to (even start to) explore log files? 1) How do students behave during assessment? à How long do students interact with a task? à Does -me- on- task vary across countries and subgroups? 2) Can we iden7fy good behavior? à Are students applying op-mal strategies? à Can we explain final performance through behavior? 3) Can we iden7fy different levels of low performance? à Where do unsuccessful students fail? à Where should interven-ons be heading? 19

1) How do students behave during assessment? à How long do students interact with a task? à Does -me- on- task vary across countries and subgroups? 20

Broad varia7on of 7me! - Time on task across students M=155.39 SD=85.16 21

No large difference of 7me on task by item score! correct incorrect 22

How do students behave during assessment? - Broad varia7on among students à Process data highlight heterogeneous behavior - BUT: No clear rela7on to performance à No easy solu-on for explana-ons of performance on an individual level 23

2) Can we iden-fy good behavior? à Are students applying op-mal strategies? à Can we explain final performance through behavior? 24

25

Applica7on of VOTAT? Strong connec7on to performance! no VOTAT Climate Control Q1 incorrect correct Total 42.2% 9.7% 51.9% VOTAT 6.9% 41.1% 48.1% Total 49.1% 50.9% 100.0% 26

Can we predict students problem solving performance? - Correla7on (7me on task, performance) = -.012 - Correla7on (VOTAT, performance) =.667 à Successful students take the same 7me but use VOTAT à Stable rela7ons across countries 27

3) Can we explain low performance? à Where do unsuccessful students fail? à Where should interven-ons be heading? 28

Top Control Temperature Central Control Humidity Bottom Control 29

How consistently did students apply VOTAT? 50 40 All countries Percent 30 20 41.1 27.3 10 incorrect correct 0 10.6 4.2 2.5 4.3 3.1 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 7.0 isolated varia7on of all input variables (VOTAT) 30

How consistently did students apply VOTAT? 50 40 Singapore Percent 30 20 63.3 10 incorrect correct 0 11.2 6.6 2.9 2.7 2.2 4.7 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 6.4 isolated varia7on of all input variables (VOTAT) 31

How consistently did students apply VOTAT? 50 40 Brazil Percent 30 20 45.5 incorrect correct 10 0 6.3 12.9 no isolated varia7on isolated varia7on of 1 input variable 1.7 3.9 1.8 4.8 isolated varia7on of 2 input variables 23.1 isolated varia7on of all input variables (VOTAT) 32

How consistently did students apply VOTAT? 50 40 Hungary Percent 30 20 32.5 34.2 10 incorrect correct 0 2.6 9.4 no isolated varia7on isolated varia7on of 1 input variable 5.8 4.5 5.5 5.5 isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 33

Differences in behavior of unsuccessful students à Different training needs Subgroups: - No isolated varia7on at all Ø Lacking understanding of isolated varia7on? - Isolated varia7on, but not for all input variables Ø Lacking understanding why complete varia7on is necessary? - Complete VOTAT, wrong solu7on Ø Lacking applica7on of available informa7on? à Process data as point of departure for interven7ons! 34

Agenda What was Problem Solving in PISA 2012? What did we do? The ra7onale Process data in PISA 2012 What did we learn? The results What s ahead? Implica7ons 35

Main results 1) How do students behave during assessment? à Lots of varia@on across students à Systema@c differences between countries 2) Can we iden@fy good behavior? à Iden@fica@on of op@mal strategies possible à Excellent predic@on of final performance 3) Can we explain low performance? à Iden@fica@on of points of failure possible à Point of departure for interven@ons à A lot of insights thanks to process data! 36

Difficult to look into students minds, but instead from the What to the How Introducing a new layer of understanding 37

Applica7ons: Repor7ng in large- scale assessments A wealth of new research endeavors Connec7on to educa7onal policy 38

When should we meet again? Gartner s hype cycle for emerging technologies PISA 2012 PISA 2015 PISA 2018 39

Thanks to: Fonds Nationale de la Recherche Luxembourg & OECD for funding this work Francesco Avvisati, Katinka Hardt, Jonas Neubert, Judit Pal, Pablo Zoido TJA fellowship programme Contact Dr. samuel.greiff@uni.lu Research Area Group "Computer- Based Assessment 11, Porte des Sciences 4366 Esch, Luxembourg Phone: +352-466644- 9245 40

Questions & discussion

How consistently did students apply VOTAT? 70 60 50 All countries Percent 40 30 20 41.1 incorrect correct 10 0 27.3 10.6 4.2 2.5 4.3 3.1 7.0 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 44

How consistently did students apply VOTAT? 70 60 50 Singapore Percent 40 30 63.3 20 incorrect correct 10 0 11.2 6.6 2.2 6.4 2.9 2.7 4.7 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 45

How consistently did students apply VOTAT? 70 60 50 USA Percent 40 30 53.0 20 incorrect correct 10 0 16.1 10.1 2.2 1.8 3.0 3.3 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 10.4 isolated varia7on of all input variables (VOTAT) 46

How consistently did students apply VOTAT? 70 60 50 Canada Percent 40 30 52.9 20 incorrect correct 10 0 18.0 10.4 2.2 2.9 5.0 2.3 6.4 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 47

How consistently did students apply VOTAT? 70 60 50 France Percent 40 30 20 48.3 incorrect correct 10 0 26.9 8.3 2.1 2.0 2.3 4.3 5.8 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 48

How consistently did students apply VOTAT? 70 60 50 Germany Percent 40 30 20 43.5 10 23.2 incorrect correct 0 11.1 1.4 0.1 5.6 4.0 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 11.1 isolated varia7on of all input variables (VOTAT) 49

How consistently did students apply VOTAT? 70 60 50 Spain Percent 40 30 incorrect correct 20 10 0 26.4 16.4 4.5 2.1 1.9 5.4 6.1 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 37.2 isolated varia7on of all input variables (VOTAT) 50

How consistently did students apply VOTAT? 70 60 50 Hungary Percent 40 30 20 32.5 34.2 incorrect correct 10 0 9.4 2.6 5.8 4.5 5.5 5.5 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables isolated varia7on of all input variables (VOTAT) 51

How consistently did students apply VOTAT? 70 60 50 Brasil Percent 40 30 20 45.5 incorrect correct 10 0 12.9 6.3 1.7 3.9 1.8 4.8 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 23.1 isolated varia7on of all input variables (VOTAT) 52

How consistently did students apply VOTAT? 70 60 50 Bulgaria Percent 40 30 20 52.0 incorrect correct 10 0 14.5 6.6 2.3 4.5 1.6 3.3 no isolated varia7on isolated varia7on of 1 input variable isolated varia7on of 2 input variables 15.1 isolated varia7on of all input variables (VOTAT) 53