HKPISA Results from HKPISA 2015 Collaborative Problem Solving: Performance of Hong Kong students in PISA 2015 Esther Sui-chu HO Director Hong Kong Centre for International Student Assessment 21 November 2017 The Chinese University of Hong Kong
Computer-based Assessment of Collaborative Problem Solving in PISA 2015 In PISA 2015, 52 countries and economies participated in a computer-based assessment (CBA) of students Collaborative Problem Solving competency PISA 2015: The first international test of students ability to work with others to solve problems 2
Overview Collaborative Problem Solving: Definition and test design in PISA 2015 Quality: Overall performance of Hong Kong students in computer-based assessment of Collaborative Problem Solving Equality: Distribution of CBA Collaborative Problem Solving performance by socio-economic status, gender and immigrant status Attitudes towards Collaborative Problem Solving Factors related to Collaborative Problem Solving performance and attitudes 3
I. Definition of Collaborative Problem Solving the capacity of an individual to effectively engage in a process whereby two or more agents attempts to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution. (OECD, 2017: PISA 2015 Assessment and Analytical Framework) 4
Definition of Collaborative Problem Solving In addition to individual problem solving competencies, there are 3 competencies specific to Collaborative Problem Solving: 1) Establishing and maintaining shared understanding 2) Taking appropriate action to solve the problem 3) Establishing and maintaining team organisation 5
Test Design Test units are interactive scenarios that students must work through while interacting with programmed computer agents Students may be asked to: Select one response out of possible options while in a conversation with the computer agent; Provide a solution to a problem using information gathered with the other agents, by clicking on a region in the visual display area Students actions will change the state of the problem 6
Test Design Chat space Task space Sample screenshot of a test unit 7
Test Design PISA 2015 includes 6 units of Collaborative Problem Solving, with a total of 117 items Sample items can be found in OECD/PISA website: Released Field Trial item: http://www.oecd.org/pisa/pisaproducts/pisa2015- Released-FT-Cognitive-Items.pdf PISA in Focus: http://dx.doi.org/10.1787/f21387f6-en 8
PISA 2015 Top 10 Countries/Economies in CBA Collaborative Problem Solving Country/Economies Mean S.E. Singapore 561 (1.2) Japan 552 (2.7) Hong Kong-China 541 (2.9) Korea 538 (2.5) Canada 535 (2.3) Estonia 535 (2.5) Finland 534 (2.6) Macao-China 534 (1.2) New Zealand 533 (2.4) Australia 531 (1.9) 10
Performance of Hong Kong Students in CBA Collaborative Problem Solving Attain a mean score of 541 Rank 3rd (3rd-7th) among the 52 participating countries/economies Of the top 10 countries/economies, Hong Kong students perform: Significantly worse than Singapore (561) and Japan (522) Not significantly different from Korea (538), Canada (535), Estonia (535) and Finland (534) Significantly better than Macao (534), New Zealand (533) and Australia (531) 11
% of Students Distribution of Students at Each Proficiency Level of CBA Collaborative Problem Solving 45 40 35 Hong Kong OECD Average 36.2 33.6 39.7 30 27.8 25 22.4 20 15 11.7 13.0 10 5 1.9 5.7 7.9 0 Below Level 1 Level 1 Level 2 Level 3 Level 4 Proficiency Level Among the five levels of CBA Collaborative Problem Solving scale, level 4 is the top level while below level 1 is the lowest level There are more high achievers (attaining level 3 and 4) in Hong Kong than in OECD countries 12
Percentage of Students at Level 4 in CBA Collaborative Problem Solving in Top 10 Countries/Economies At the upper end, 13.0% of Hong Kong students reach level 4, outperforming the OECD average of 7.9%, but being outperformed by Singapore (21.4%) and Japan (14.0%) 13
% of students Percentage of Students at Level 1 and below in CBA Collaborative Problem Solving in Top 10 Countries/Economies 25 20 18.5 18.1 19.7 19.9 15 10 11.4 10.1 13.7 12.9 15.2 14.9 5 0 At the lower end, 13.7% of Hong Kong students perform at level 1 and below, which is lower than the OECD average of 28.1% but higher than Japan (10.1%) and Singapore (11.4%) 14
Collaborative Problem Solving Mean Score 1. CBA Collaborative Problem Solving Performance by Socio-Economic Status China (B-S-J-G) Chinese Taipei Korea Hong Kong Japan Macao 640 Singapore Finland Estonia Canada New Zealand Australia Level 4 600 560 Hong Kong (14) Macao (8) Singapore (33) Level 3 520 480 China (B-S-J-G) (35) 440 Level 2 400-3 -2-1 0 1 2 3 Index of Economic, Social and Cultural Status (ESCS) Socio-economic status of HK students has a relatively small impact on their performance (14) compared with OECD average (30) 16
Mean Collaborative Problem Solving Score OECD average = 7.9 CBA Collaborative Problem Solving Performance and the Impact of Socio-Economic Status 600 Strength of the relationship between performance and socio-economic status is above the average Strength of the relationship between performance and socio-economic status is not statistically significantly different from the average Strength of the relationship between performance and socio-economic status is below the average 550 500 450 400 OECD average Peru China (B-S-J-G) Hungary Bulgaria Colombia Belgium France Uruguay Canada Singapore Finland Japan Chinese Taipei Korea Hong Kong-China New Zealand Germany Estonia United States Australia Macao-China Austria Denmark Sweden United Kingdom Czech Republic Netherlands Norway Iceland Slovenia Spain Portugal Latvia Luxembourg Italy Israel Croatia Lithuania Russian Federation Chile Greece Slovak Republic Malaysia Costa Rica Cyprus Thailand United Arab Emirates Mexico Turkey Montenegro Brazil Tunisia 350 25 20 15 Percentage of variation in performance explained by socio-economic status (R2 x 100) Hong Kong belongs to the group of high performance/ low socioeconomic impact countries/economies (upper right quadrant) 10 5 17 0
2. Gender Difference in CBA Collaborative Problem Solving Performance (Top 10 Countries/Economies) Country/Region Mean score Boys (B) Girls (G) Difference (B - G) # S.E. Mean score S.E. Score dif. Singapore 552 (1.7) 572 (2.1) -20 (2.9) Japan 539 (3.6) 565 (2.6) -26 (3.7) Hong Kong-China 523 (3.7) 559 (3.4) -36 (4.4) Korea 522 (3.5) 556 (3.3) -33 (4.4) Canada 516 (2.8) 555 (2.4) -39 (2.6) Estonia 522 (2.9) 549 (2.7) -27 (2.8) Finland 511 (3.2) 559 (3.0) -48 (3.6) Macao-China 515 (1.9) 553 (2.0) -38 (2.9) New Zealand 513 (3.2) 553 (3.0) -41 (3.8) Australia 511 (2.5) 552 (2.5) -41 (3.1) OECD Average 486 (0.6) 515 (0.5) -29 (0.6) Values that are statistically significant are indicated in bold. # The minor discrepancy in the difference is due to rounding of numbers. HK girls advantage (36) is greater than the average gender gap of OECD (29) 18 S.E.
Collaborative Problem Solving Mean Score 3. CBA Collaborative Problem Solving Performance by Immigrant Status No sig. diff. 560 547 540 536 529 520 505 Sig. diff. 500 482 Sig. diff. 480 460 459 440 420 400 Native students Hong Kong-China Second-generation immigrants Sig. diff. OECD Average First-generation immigrants Second-generation immigrants: Students who are born in the country of assessment but both of their parents are foreign-born First-generation immigrants: Students whose parents and they themselves are not born in the country of assessment 19
1. Attitudes towards Collaborative Problem Solving Valuing relationships index ( 重視關係 ) Altruistic attitude held by a student when engaging in collaborative activities not for his or her own benefit Students were asked to report to what extent they disagreed or agreed with each of the 4 statements about themselves: 1) I enjoy considering different perspectives. 2) I am a good listener. 3) I take into account what others are interested in. 4) I enjoy seeing my classmates be successful. Coded as: 1 for Strongly disagree ; 2 for Disagree ; 3 for Agree and 4 for Strongly agree. Items are coded and scaled such that higher scores on this index mean a higher level of valuing relationships. 21
Attitudes towards Collaborative Problem Solving Valuing teamwork index ( 重視團隊 ) Emphasis put on what teamwork, as opposed to working alone, can produce Students were asked to report to what extent they disagreed or agreed with each of the 4 statements about themselves: 1) I enjoy cooperating with peers. 2) I find that teams make better decisions than individuals. 3) I find that teamwork raises my own efficiency. 4) I prefer working as part of a team to working alone. Coded as: 1 for Strongly disagree ; 2 for Disagree ; 3 for Agree and 4 for Strongly agree. Items are coded and scaled such that higher scores on this index mean a higher level of valuing teamwork. 22
% of students Attitudes towards Collaborative Problem Solving of Hong Kong Students (% of Agree or Strongly Agree) 100.0 90.0 80.0 70.0 91.7% 89.8% 89.7% 86.8% 87% 86.3% 87.8% 84.8% 87% 84.5% 80.2% 73.5% 76.9% 69.8% 71% 67.3% 60.0 50.0 40.0 30.0 20.0 10.0 0.0 I enjoy considering different perspectives I am a good listener I take into account what others are interested in I enjoy seeing my classmates be successful I enjoy cooperating with peers I find that teams make better decisions than individuals I find that teamwork raises my own efficiency I prefer working as part of a team to working alone Valuing Relationships Valuing Teamwork Hong Kong OECD Average 23
Indices of Attitudes towards Collaborative Problem Solving of Students in East Asian Societies OECD average = 0.00 Hong Kong students indices of attitudes towards Collaborative Problem Solving are similar to OECD average 24
Latvia Slovak Republic Czech Republic Japan Russia Poland Italy Netherlands Finland Montenegro Slovenia Belgium Croatia Peru France United Kingdom Macao (China) Bulgaria Hungary Ireland Estonia Iceland Turkey New Zealand OECD average Luxembourg Denmark Hong Kong (China) Norway Greece Colombia Brazil Sweden Korea Chile B-S-J-G (China) Australia Germany Switzerland Lithuania Canada Austria Tunisia Qatar Uruguay Mexico Thailand United States Israel Spain Chinese Taipei Dominican Republic United Arab Emirates Portugal Costa Rica Singapore Value relationships more Mean index Index of Valuing Relationships, by Gender Table V.5.4a 0.60 0.40 Girls are more likely to value relationships Boys Girls 0.20 0.00-0.20-0.40-0.60
Netherlands Norway Finland Russia Iceland Slovak Republic Montenegro Latvia Sweden Japan Estonia Israel Belgium Denmark Bulgaria Turkey Poland United Kingdom Macao (China) Hungary Czech Republic Luxembourg Slovenia OECD average Ireland Hong Kong (China) Italy Australia Canada Peru France New Zealand United States Spain Germany Greece Qatar Austria Korea Brazil Uruguay Colombia Switzerland Chile Croatia Mexico Singapore Portugal Lithuania Costa Rica B-S-J-G (China) Chinese Taipei Thailand Tunisia United Arab Emirates Dominican Republic Value teamwork more Mean index Index of Valuing Teamwork, by Gender Table V.5.4b 0.60 0.50 0.40 Boys are more likely to value teamwork Boys Girls 0.30 0.20 0.10 0.00-0.10-0.20-0.30-0.40
Collaborative Problem Solving Mean Score Relationship between Attitudes towards Collaborative Problem Solving and Student Performance in Hong Kong 555 550 548 548 549 545 540 537 539 538 538 535 532 530 525 520 Index of Valuing Relationships Index of Valuing Teamwork Bottom quarter Second quarter Third quarter Top quarter HK students attitude of valuing relationships has a positive relationship, but their attitude of valuing teamwork has a negative relationship with Collaborative Problem Solving performance 27
Collaborative Problem Solving Mean Score 1. ICT Resources in Family and Student Performance in Hong Kong Sig. diff. 550 545 540 530 520 510 505 500 490 480 With access to computers at home (93%) Without access to computers at home (7%) HK students having access to computers at home outperform significantly those without by 40 points in CBA Collaborative Problem Solving 29
2. Online Activities in and outside School In Hong Kong, moderate users of ICT (once or twice a week or a month) outside school tend to have better performance of CBA Collaborative Problem Solving than frequent users (every day) or non-users However, use of ICT in school has a negative relationship with Collaborative Problem Solving performance Students who need to use ICT in school may be the disadvantaged students who cannot afford ICT facilities at home 30
Relationship between Online Activities in and outside School and CBA Collaborative Problem Solving Performance (Hong Kong) Browsing the Internet for schoolwork (e.g. for preparing an essay or presentation) 495 536 542 557 Browsing the Internet to follow up lessons (e.g. for finding explanations) 519 533 547 553 Obtaining practical information from the Internet 525 538 550 554 Never or hardly ever Once or twice a month Reading news on the Internet (e.g. current affairs) 519 540 550 547 Once or twice a week Almost every day/every day Using email at school 488 530 550 551 Doing homework on a school computer 487 525 552 551 440 460 480 500 520 540 560 580 Mean Collaborative Problem Solving Score 31
Singapore -18 Japan 14 Hong Kong (China) -34 Korea -24 New Zealand -29 Estonia -42 Germany -22 Finland -29 Denmark -21 Sweden -39 Macao (China) United Kingdom -14 Australia 14 Chinese Taipei Czech Republic -46 Netherlands -24 Austria -30 Portugal -50 B-S-J-G (China) -33 OECD average -29 Luxembourg -46 Slovenia -28 Iceland -25 Latvia -51 Belgium -12 France -19 Spain Italy -40 Russia -40 Israel -60 Hungary -48 Greece -68 Croatia -38 Lithuania -53 Bulgaria -65 Slovak Republic -35 Chile -36 Uruguay -47 Thailand -29 Costa Rica -19 Peru -33 Colombia -16 Brazil -24 Mexico Using ICT and Digital Devices in School and Collaborative Problem Solving Performance Figure V.3.12 600 560 Mean score Index of ICT use at school: Top quarter Third quarter Second quarter Bottom quarter In most countries/ economies, use of ICT in school has a negative relationship with Collaborative Problem Solving performance 520 480 440 400 Performance difference between top and bottom quarters of the index of ICT use at school
3. Learning Environment in Science Class PISA asked students about how often they engage in communication-intensive activities such as explaining one s ideas in science class; spending time in the laboratory doing practical experiments; arguing about science questions; and taking part in class debates about investigations. Positive relationship between these activities and attitudes towards collaboration
Percentage-point difference Figure V.6.9 Student Interaction in Science Class and Attitudes towards Collaboration (OECD Countries) 3 2 After accounting for gender and students' and schools' socio-economic profile Before accounting for gender and students' and schools' socio-economic profile Students who report that more communication-intensive activities take place in science class have more positive attitudes towards collaboration 1 0 I am a good listener I enjoy seeing my classmates be successful I take into account what others are interested in I enjoy considering different perspectives I prefer working as part of a team to working alone I find that teams make better decisions than individuals I find that teamwork raises my own efficiency I enjoy cooperating with peers Items comprising the index of valuing relationships Items comprising the index of valuing teamwork
Perecntage-point difference Student Interaction in Science Class and Attitudes towards Collaboration (Hong Kong) Figure V.6.9 2.0 1.5 After accounting for gender, and student and school socio-economic profile Before accounting for gender, and student and school socio-economic profile 1.9 Students who report that more communication-intensive activities take place in science class have more positive attitudes towards collaboration 1.0 0.7 0.6 0.6 0.8 1.0 1.0 0.7 0.5 0.0 I am a good listener I enjoy seeing my classmates be successful I take into account what others are interested in I enjoy considering different perspectives I prefer working as part of a team than working alone I find that teams make better decisions than individuals I find that teamwork raises my own efficiency I enjoy cooperating with peers Items comprising the index of valuing relationships Items comprising the index of valuing teamwork
4. Disciplinary Environment in School PISA asked students about Being Bullied: e.g. being threatened by other students; Truancy: e.g. skipping a school day, some classes or arriving late for school. Negative relationship between these disciplinary problems and CPS performance and attitudes towards collaboration
Change in index Student Truancy and Attitudes towards Collaboration (OECD Countries) Figure V.6.7 After accounting for gender and students' and schools' socio-economic profile Before accounting for gender and students' and schools' socio-economic profile 0.00 Index of valuing relationships Index of valuing teamwork -0.05-0.10-0.15-0.20-0.25 Students who play truant are more likely to show negative attitudes towards collaboration -0.30 Skipping a whole day of school Skipping some classes Arriving late for school Skipping a whole day of school Skipping some classes Arriving late for school
Difference Difference Student Truancy and Attitudes towards Collaboration (Hong Kong) Figure V.6.7 Before accounting for gender and students' and schools' socio-economic profile After accounting for gender and students' and schools' socio-economic profile Index of valuing relationships Index of valuing teamwork 0.05 0.05 0.00 0.00-0.05-0.05-0.10-0.10-0.15-0.15-0.20-0.20-0.25-0.25-0.30-0.30-0.35 Skipping a whole day of school Skipping some classes Arriving late for school Skipping a whole day of school Skipping some classes Arriving late for school -0.35 Students who play truant are more likely to show negative attitudes towards collaboration
Luxembourg Greece Singapore United States Norway Macao (China) Spain Canada New Zealand Denmark Chinese Taipei Hong Kong (China) Czech Republic Finland Estonia United Arab Emirates Australia Portugal B-S-J-G (China) Thailand Sweden OECD average Slovenia Peru Iceland Latvia Chile Slovak Republic Brazil Montenegro Austria Colombia Croatia Japan Bulgaria Lithuania United Kingdom Uruguay Russia Belgium Netherlands Hungary Mexico Turkey Germany Tunisia France Costa Rica Korea Score-point difference Figure V.7.3 Being Bullied: Students Being Threatened by Other Students and Collaborative Problem Solving Performance 45 40 35 30 Change in score after accounting for students' and schools' socio-economic profile At the school level At the student level In most countries, students score higher when they reported not being threatened by other students 25 20 15 10 5 0-5
Korea Chinese Taipei Germany Netherlands Montenegro Portugal Russian Iceland Brazil Luxembourg Costa Rica France Switzerland Dominican Hungary Spain Turkey Chile Greece Ireland Uruguay Belgium Poland China (B-S-J-G) United States Austria Croatia Peru Slovak Republic Macao-China OECD Average Denmark Colombia Lithuania Norway Slovenia Finland Estonia United Kingdom Singapore Thailand Mexico Canada Cyprus Malaysia Australia Sweden New Zealand Japan Czech Republic United Arab Hong Kong-China Qatar Bulgaria Tunisia Latvia Being Bullied: Percentage of Hong Kong is Higher Than OECD Average 30 25 20 15 10 5 0 2 2.3 Percentage of students reported physically bullied in school (PISA2015) 23.4 24.5 25.6 5.8 6.3 6.9 7 7.4 7.5 8.3 8.5 8.6 8.7 9.1 9.2 9.5 9.7 9.9 10 10.110.310.410.510.5 10.710.8 11 11 11.511.611.611.912.7 12.7 12.712.8 13.5 13.5 14 14.714.914.915.115.3 15.315.4 16 16.8 18.118.518.819.420.2 21.1 % reported get hit or pushed around by other students 40
5. Parent Factors Parental Involvement Science activity Academic communication Social communication Emotional Support Emotional support (student report) Emotional support (parent report) 41
Items for Science Activity, Academic and Social Communication PISA asked parents about their child s engagement in science activities at about age 10 such as: Watching TV programmes about science Reading books on scientific discoveries PISA asked parents about their academic communication with their child. Examples are: Helping my child with his/her science homework Asking how my child is performing in science class PISA asked parents about their social communication with their child. Examples are: Eating dinner with my child around a table Spending time just talking to my child 42
Collaborative Problem Solving Mean Score Parental Involvement Indices and Collaborative Problem Solving Performance (Hong Kong) 560 Bottom quarter Second quarter Third quarter Top quarter 550 540 542 549 543 542 550 545 545 540 551 530 533 528 530 520 510 Previous science activities Current academic communication Current social communication Parental social communication is the most important! 43
Items for Emotional Support PISA asked parents about their emotional support for their child. Examples are: I am interested in my child s school activities I encourage my child to be confident PISA asked students about their parents emotional support for them. Examples are: My parents are interested in my school activities My parents encourage me to be confident 44
Collaborative Problem Solving Mean Score Parental Support Indices and Collaborative Problem Solving Performance (Hong Kong) 560 Bottom quarter Second quarter Third quarter Top quarter 550 547 545 550 546 540 535 537 540 530 530 520 Parent Emotional support (student report) Parent Emotional support (parent report) Parental emotional support is the most important! 45
Parent Factors and Collaborative Problem Solving Performance (Hong Kong) 46
Parental Factors and Attitudes towards Collaboration (Hong Kong) Index of valuing relationships Index of valuing team work Parental emotional support (student report) is the most important! 47
Conclusions 1) Hong Kong students performance in CBA Collaborative Problem Solving Similar to their performances in CBA science, reading and mathematics in PISA 2015, Hong Kong students performance in Collaborative Problem Solving are among the top 10 of all participating countries and economies 2) Low impact of SES on CBA Collaborative Problem Solving performance Socio-economic status of Hong Kong students has a relatively small impact on their Collaborative Problem Solving performance Among all participating countries and economies, Hong Kong belongs to the group of high performance/ low socio-economic impact countries and economies 48
Conclusions 3) Gender gap in CBA Collaborative Problem Solving Hong Kong girls outperform boys in Collaborative Problem Solving 4) Immigrant students disadvantage in CBA Collaborative Problem Solving Similarly to CBA science, reading and mathematics, immigrant students in Hong Kong perform significantly more poorly than native students in Collaborative Problem Solving 5) Attitudes towards Collaborative Problem Solving Hong Kong students attitudes of valuing relationships and valuing teamwork are near to international average Students attitude of valuing relationships has a positive relationship whereas attitude of valuing teamwork has a negative relationship with Collaborative Problem Solving performance 49
Conclusions 6) ICT resources in Family A great majority of Hong Kong students (93%) have access to computers and internet at home Students having access to computers at home perform better than those without in Collaborative Problem Solving 7) Online activities Moderate users of ICT at home perform better than frequent users or nonusers in Collaborative Problem Solving 8) Learning environment in class and in school Positive relationship between interactive science activities and attitudes towards collaboration Negative association between truancy and attitudes towards collaboration Negative association between bullying and Collaborative Problem Solving performance 9) Family factors Social communication and emotional support have positive relationships with Collaborative Problem Solving performance and attitudes 50
HKPISA Thank you! For further information: OECD/PISA Website: http://www.oecd.org/pisa/ Email: edu.pisa@oecd.org HKCISA Centre Website: www.fed.cuhk.edu.hk/~hkcisa Email: estherho@cuhk.edu.hk