Student Collaborative Problems Solving in a Scientific Inquiry Learning Environment

Student Collaborative Problems Solving in a Scientific Inquiry Learning Environment Abstract The study investigated Collaborative Problems Solving (CPS) processes in the context of inquiry-based learning in high-school biology. Findings show that the effectiveness of CPS in an inquiry-oriented, technology-rich learning environment varies considerably as function of the specific sub-tasks students engage in during the project, the type of collaboration (faceto-face dyads versus virtual teams), and the specific team students belong to. Findings also provided evidence for the usefulness of student self-evaluations in assessing CPS. Teams differ considerably in CPS effectiveness, and team collaborative quality was associated with the quality of learning outcomes. A profile of collaboration across different tasks and teams may inform theory by shedding light on potential team x situation interactions, and provide useful instructional feedback to teachers. Objectives The main purpose of this study was to examine Collaborative Problem Solving (CPS) processes that arise when students engage in an authentic scientific inquiry. We sought to describe different facets of collaborative learning and performance, during different stages of an inquiry project, to investigate between-team variability in CPS and its correlation with learning outcomes. The empirical findings of the study should inform the emerging pedagogical theory of CPS in the rich context of project-based learning, when students collaborate for an extended period of time to investigate a complex scientific questions. An additional goal of the study was to design and test the reliability and validity of an assessment instrument based on student reflective self-evaluation of their collaborative experiences. Theoretical framework As education systems evolve to encompass an ever-expanding range of learning and teaching goals, much of the attention is directed at a set of skills, collectively known as "21st Century skills. Several typologies have been offered to map out the plethora of cognitive (basic and higher-order), interpersonal, and intrapersonal skills assembled under this broad category (see, e.g., National Research Council, 2012). Common to all the frameworks is the emphasis on Collaborative problem solving (CPS) as especially important for adaptive functioning in current and future social and workplace environments. We draw on the ATC21S (Hesse, 2015) and the PISA (OECD, 2013) conceptual frameworks to define collaboration as a shared effort of two or more partners working towards a common 1

goal. Problem solving, individually or in a group, entails the perception of discrepancy between a current state and a desired goal state, and then devising and implementing a strategy to narrow the discrepancy and reach the desired goal. Thus, CPS refers to a combination of collaborative processes geared to achieve specified goals that include: establishing and maintaining shared understandings among team members; designing cooperative activities, enacting plans and monitoring progress; establishing and maintaining team organization (roles and responsibilities, rules of engagement, and constructive feedback mechanisms). CPS is a powerful learning experience, especially when problems are complex and require multiple perspectives or distributed expertise (Fawcett, & Garton, 2005; Hattie, 2009; Roseth et. al., 2006). Teams may form rich learning environments by providing social comparisons, cognitive restructuring, giving feedback, emotional support and deliberate practice. On the other hand, team performance may be compromised when division of labor is haphazard, sub-goals are not achieved due to tardiness of team members or the absence of effective progress monitoring, and communication breakdowns. It is not surprising that team composition is one of the key factors in the success or failure of CPS efforts (Dillenbourg, 1999; Rosen, & Rimor, 2009). Meaningful learning via collaborative interactions rarely emerges spontaneously and requires careful structuring and support. There is a growing emphasis on project-based and inquiry-oriented curriculum and instruction that incorporate critical thinking, problem solving, self-regulation and collaboration skills (Darling-Hammond 2011, National Research Council, 2011). Such learning environments require students to work together to achieve a team goal, such as a final report, and in school-based contexts, are often integrated into domain-specific courses of study. Increasingly, collaborative environments become technology rich and require virtual collaboration (Davis, Fidler & Gorbis, 2011). The current study aims contribute to the theory and practice of CPS in inquiry-oriented, technology-rich educational context, by offering a systematic analysis of variability in CPS processes and performance among student teams engaged in complex scientific inquiry projects in biology. Methodology and data sources 364 12th grade students from 19 high-schools participated in a collaborative science inquiry project, as part of their matriculation exam in biology. The Ministry of Education provides for an optional performance assessment of inquiry tasks that counts towards a final grade, comprised (in addition to score on the inquiry task) of a teacher overall grade in the subject and a score on an external standardized test (with different weights assigned to the different components.) Schools, teachers, and students opted to participate in this experimental project that introduced for the first time between-school collaborations. Participating schools varied in terms of ethnic composition, socio-economic status, and religiosity. 2

Students were paired (based on preferences and teacher considerations) into dyads within each classroom, and two dyads from different schools (usually matched on background characteristics) formed collaborative teams (for logistic reasons, three schools were not paired and teams were comprised of two dyads from different classrooms within the school). A large educational NGO provided the infrastructure for the project laboratory equipment and materials, and a specially designed online portal (on a Moodle platform). The portal designated a work space for each dyad and team that included scientific information (mainly articles) on a range of biology topics, links to external resources, and links to structured Google Docs for collaborative writing. Students conducted a full scientific research project: dyads chose and gathered information on topics for research, formulated initial research questions and performed a pilot check of procedures and instrumentation. Teams coordinated the final questions (different dyads could, for example, explore different aspects of a common biological phenomenon) and shared preliminary results to generate the final research design. After documenting and sharing their procedures via video clips, dyads obtained and shared their final results and wrote together an integrative discussion, comparing and interpreting their findings vis a vis the research questions. Teams then produced a final collaborative report of the entire inquiry. Students and teachers met once in a conference at the beginning of the year (Sept) for an introduction to the project, team pairing, and initial brainstorming. The rest of the teamwork was conducted online and concluded (in March) with a final paper and an oral examination (conducted via videoconferencing). While dyads interacted during class time, dyad and team partners communicated via WhatsApp and comments within the shared documents. Pairs of teachers collaborated to monitor and support dyad and team work throughout the project. Teachers provided evaluations of the quality of teamwork (on a 0-100 scale), based on their observations and evidence of collaborative efforts (e.g. mutual comments in shared documents, synthesis of inquiry information, etc.). Each teacher independently evaluated the collaborative teamwork performed by their own students. Teachers also graded by consensus each team's final paper, guided by a standardized, detailed scoring rubric (on a 0-100 scale). Students were asked to respond to a feedback questionnaire, after they completed their final papers. In addition to items relating to various logistical aspects of the project, students evaluated each task they performed in dyads and teams (see Figure 1 for details) and rated (on a 1-5 Likert scale) the extent to which collaboration on each task was successful and effective. Students also evaluated the overall quality of collaboration in the project by responding to the following items, representing key aspects of CPS, based on the PISA framework (OECD, 2013): The team reached a good understanding of the tasks We were able to share information efficiently Team division of labor was fair and effective Team members gave useful feedback during collaboration The team managed to resolve differences successfully The team managed to finish tasks in the prescribed deadlines All team members participated actively in discussions and decisions 3

Psychometric and substantive findings are presented and discussed below. Results and conclusions Collaborative tasks in dyads and teams Average student self-evaluation of the success and effectiveness of the various collaborative tasks, leading to the final paper, are presented in Figure 1. Student evaluations varied considerably across different tasks, and dyadic tasks were consistently rated higher than team tasks. The profile of tasks show that team collaboration was especially challenging in the final stages of the project, when dyads across schools shared their results and conclusions, and produced shared reports, integrating and summarizing their insights. Exploratory factor analysis (with Varimax rotation) of task evaluations indicated two distinct factors, explaining 66% of the variance. All dyadic tasks were highly loaded one factor and team tasks on the other. Separate Dyad (α=.88) and Team (α=.91) scores were computed by averaging item responses for each factor. Overall, students evaluated the quality of collaboration as better in the dyadic tasks (M=4.07, SD=0.76) than the team tasks (M=3.63, SD=1.01) [t(360)=8.97; p<.001; Cohen's d=.51]. They also reported (in two additional items) that they enjoyed the dyadic collaboration with their classmates (M=4.04, SD=1.15) more than the between-school team work (M=3.26, SD=1.40) [t(348)=10.22; p<.001; Cohen's d=.61]. Team collaboration and performance Exploratory factor analysis (with Varimax rotation) of the 7 evaluative items of the overall quality of teamwork indicated a single factor, explaining 66% of the variance. A Collaboration score (α=.91) was computed by averaging item responses. A multilevel model with team as level-2 random effects factor showed that 26% of the variance in student responses, and 25% of the variance in teacher evaluations of the quality of team collaboration, could be attributed to between-team differences. These results indicate a sizeable team effect on student and teach evaluations of the quality of collaboration collaboration processes vary considerably across teams. In order to further investigate the between-team variance in collaboration and performance, student evaluations of overall and specific team task collaboration, and teacher overall team collaboration evaluations were aggregated (by averaging individual team members' data) to the team level. Complete student and teacher data were available for 57 teams. Table 1 presents the intercorrelations among collaboration and performance indicators. Student overall and specific team tasks evaluations were highly correlated; teacher evaluations were substantially correlated with both student overall and task-specific evaluations; and, most importantly, team performance (final paper grade) was substantially correlated with teacher and student evaluations of team collaboration (but not with task-specific evaluations). These results demonstrate the convergent validity of student and teacher reports of the quality of collaborative teamwork. Moreover, they lend support to the hypothesis that the 4

quality of collaboration in teams enhances performance in authentic scientific inquiry teams that collaborate more effectively produce better learning outcomes. Discussion The current study shows that the effectiveness of CPS in an inquiry-oriented, technology-rich learning environment varies considerably as function of the specific sub-tasks students engage in during the project, the type of collaboration (face-to-face dyads versus virtual teams), and the specific team students belong to. It also provided evidence for the usefulness of student self-evaluations in assessing CPS. Previous studies have documented CPS processes by focusing on collaborative and problem solving processes aggregated across sub-tasks during an entire learning sequence (for example, Jahnke, 2010; Hou, 2011). Our findings underscore the need for a nuanced analysis of the dynamics of collaboration as a project progresses. Different stages of a complex inquiry project evoke different sub-tasks (e.g., formulate research questions, share results, and produce reports) that place different demands for collaboration and elicit different collaborative processes. A profile of collaboration across different tasks can inform theory by shedding light on potential team x situation interactions, and provide useful instructional feedback to teachers on tasks where more intense support is needed. In the current project, for example, sharing results across dyads and team collaborative writing emerged as more challenging than other tasks. Findings also show that within-school face-to-face dyads collaborated better than betweenschool virtual teams. As a growing body of research focuses on virtual collaborations (e.g., Rosen & Foltz, 2014), we emphasize the importance of direct interpersonal relations, partner availability, and efficiency of communication, as factors that may give face-to-face teams some important advantages. Obviously, more research and design efforts are needed to develop effective virtual tools for collaboration as well as for teacher facilitation of collaborative projects. The question remains whether a unified theory that encompasses both face-to-face and virtual CPS learning and instructional processes can emerge. Like other studies, we conclude that CPS is sensitive to group composition. This study shows that teams differ considerably in CPS effectiveness and that team collaborative quality is an important determinant of the quality of learning outcomes. It was interesting to observe that when teams collaborated effectively, final papers consistently received high grades, but when collaboration was not optimal, grades varied. This phenomenon may be due to individual team members' efforts to compensate for the breakdown of collaboration. 5

Figure 1. Student mean evaluations of collaborative tasks in dyads (D) and teams (T) Table 1. Correlations of collaboration and performance variables Variables 1 2 3 1. PAPER 2. T-COLL.43** 3. S-COLL.31*.40** 4. TASKS.25.44**.83*** Note: PAPER = Teacher grade of final team paper; T-COLL = Teacher evaluation of overall teamwork collaboration; S-COLL = Student evaluation of overall teamwork collaboration; TASKS = Student evaluation of collaboration in team tasks; N = 57 teams. * p<.05 ** p<.01 ***p<.001 6

References Darling-Hammond, L. (2011). Policy frameworks for new assessments. In P. Griffin, B. McGaw, & E. Care (Eds.). Assessment and teaching 21st century skills. Heidelberg: Springer. Davis, A., Fidler, D., & Gorbis, M. (2011) Future Work Skills 2020. Institute for the Future for University of Phoenix Research Institute. Retrieved from: http://www.iftf.org/futureworkskills2020 Dillenbourg, P. (Ed.). (1999). Collaborative learning: Cognitive and computational approaches. Amsterdam, NL: Pergamon, Elsevier Science. Fawcett, L. M., & Garton, A. F. (2005). The effects of peer collaboration on children s problem solving ability. British Journal of Educational Psychology, 75, 157 169. Hattie, J. A. C. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. London: Routledge. Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A Framework for Teachable Collaborative Problem Solving Skills. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp.37-56). Dordrecht: Springer. Hou, H. (2011). A case study of online instructional collaborative discussion activities for problem-solving using situated scenarios: An examination of content and behavior cluster analysis. Computers & Education, 56(3), 712-719. Jahnke, J. (2010). Student perceptions of the impact of online discussion forum participation on learning outcomes. Journal of Learning Design, 3(2), 27-34. National Research Council (2011). A Framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: The National Academies Press. National Research Council. (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Committee on Defining Deeper Learning and 21st Century Skills, J.W. Pellegrino and M.L. Hilton, Editors. Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. OECD (2013). PISA 2015 Collaborative Problem Solving framework. OECD Publishing. Retrieved from: http://www.oecd.org/pisa/pisaproducts/pisa2015draftframeworks.htm Rosen, Y., & Foltz, P. (2014). Assessing collaborative problem solving through automated technologies. Research and Practice in Technology Enhanced Learning, 9(3), 389-410. Rosen, Y, & Rimor, R. (2009). Using collaborative database to enhance students knowledge construction. Interdisciplinary Journal of E-Learning and Learning Objects, 5, 187-195. Roseth, C. J., Fang, F., Johnson, D. W., & Johnson, R. T. (2006, April). Effects of cooperative learning on middle school students: A meta-analysis. Paper presented at the Annual Meeting of the American Educational Research Association. San Francisco, CA. 7