AN EXAMPLE OF PROBLEM-BASED LEARNING (PBL) FROM A COLLABORATIVE AND MULTIDISCIPLINARY APPROACH

AN EXAMPLE OF PROBLEM-BASED LEARNING (PBL) FROM A COLLABORATIVE AND MULTIDISCIPLINARY APPROACH A. Baviera-Puig 1, C. Escribá-Pérez 1, J. Buitrago-Vera 1, F. Ramón-Fernández 2, L. Rivera-Vilas 1, B. Roig-Merino 1, E. Sigalat-Signes 1, J.V. Pons-Valverde 1 1 Department of Economics and Social Sciences. Universitat Politècnica de València (SPAIN) 2 Department of Planning. Universitat Politècnica de València (SPAIN) Abstract In order to bring college students to the world of business, we developed an Educational Innovation and Improvement Project at the Universitat Politècnica de València (Valencia, Spain) during the second semester of the academic year 2015/2016. In this project, students faced a real problem presented by a real company. Students developed a solution to this problem using the problem-based learning (PBL) method. Given the complexity of the task, this teaching methodology was carried out by working groups (3-4 students). All group members had to contribute to the elaboration of the final solution. Therefore, the learning process acquired a collaborative nature. We developed this project in our University in bachelor s degree in Business Administration and Management, bachelor s degree in Tourism, master s degree in Agricultural Engineering and master s degree in Construction. The PBL method is a useful tool for building a range of generic skills. However, we only focused on some of them. The aim of the project was to assess students achievement in four generic skills: Ability to think practically and apply knowledge in practical situations, Ability to analyse and resolve problems, Teamwork and leadership ability and Awareness of current business issues. We also wanted to measure the differences based on student profile as the project had a multidisciplinary approach. The assessment was different for each skill. Depending on the skill, we used a written report by group, student self-evaluation, student co-evaluation and/or observation. Using these results, PBL can give to the lecturers a way to measure the development of these skills by the students. At the end of the project, we recommend to use objective evidences. Based on the student profile, there are differences in the mean scores obtained by course but we didn t find differences by degree (bachelor or master). Keywords: Problem-based learning (PBL), collaborative, generic skills, assessment, multidisciplinary approach, company. 1 INTRODUCTION A major problem with university education is the applicability of classroom teaching to the real world. Work placements help in this area, but not all students have such an opportunity to apply their knowledge outside the classroom. The ideal scenario would be to present students in the university lecture real issues faced by real firms. This was the main challenge of the Educational Innovation and Improvement Project developed at Universitat Politècnica de València (Valencia, Spain) during the second semester of the academic year 2015/2016. In this project, students proposed solutions to a real problem presented by a real company. These solutions were developed through the teaching methodology problem-based learning (PBL), very useful tool for developing generic skills of all kinds ([1], [2], [3]). According to [1] and [3], PBL can also be used to link the university with the resolution of real problems faced by firms. Therefore, this teaching method has become an ideal tool for resolving this major challenge currently being faced by university education. PBL is a teaching methodology that organizes the process of teaching and learning through the development of projects by students ([4]). Therefore, learning occurs as a result of the effort made by the student to carry out a project. Thus, students work actively, planning, implementing and controlling Proceedings of EDULEARN16 Conference 4th-6th July 2016, Barcelona, Spain 5873 ISBN: 978-84-608-8860-4

projects with an application in the real world, going beyond the classroom of class ([5], [6], [7]). PBL is thus a student-focused method ([8]). PBL is characterized by being action-oriented. Project development becomes thus a teaching strategy that is part of the so-called active methodologies. [9] points out that the advantages of PBL are: It is interesting for the student. It becomes an incentive for the student. Allows the acquisition of a methodology of professional work. The student learns from the experience. The student develops independent learning and creative thinking. The PBL method is a useful tool for building a wide range of generic skills from the "Tuning Project" ([10]). The aim of this project was to show how PBL can be used to evaluate students attainment of four generic skills. The specific objectives of this research were to: measure the achievement of the generic skill Ability to think practically and apply knowledge in practical situations; measure the achievement of the generic skill Ability to analyse and resolve problems; measure the achievement of the generic skill Teamwork and leadership ability; measure the achievement of the generic skill Awareness of current business issues; and check for significant differences in the achievement of students enrolled in different courses at the Universitat Politècnica de València (UPV). 2 METHODOLOGY Students proposed solutions to a real life issue faced by a real firm. Students proposed these solutions following the project-based learning (PBL) method. Before beginning the process, a manager from the firm gave a presentation to introduce the problem. Students worked in teams throughout the entire process, developing a project to resolve the problem presented by the manager. Students formed groups of three or four and had two months to complete the project. After two months, the students had to submit a group written report. All group members had to contribute to the elaboration of the final solution. Therefore, the learning process acquired a collaborative nature. As this project had a multidisciplinary approach, results were evaluated first for each degree course and then between different degree courses. Results were compared across the following four courses at the UPV depending on its level: LEVEL I (1st and 2nd years of Degree): Tourism Marketing: 2nd year course in the bachelor s degree in Tourism Management. LEVEL II (3rd and 4th years of Degree): Marketing Management: 3rd year course in the bachelor s degree in Business Administration and Management. LEVEL III (Master): Agri-Food Marketing: course in the master s degree in Agricultural Engineering. Business Management in Construction: course in the master s degree in Construction. The study consisted of measuring students achievement in four generic skills. Our University, taking into account the most important standards and guidelines as well as national and international scientific literature, has listed 13 generic skills. These skills synthesize a profile that has to be acquired by all the students at the UPV. From this list, we chose four generic skills for our study. Our University has also developed a rubric for every skill depending on the level of the course. We used these rubrics when assessing the students achievement. We also combined different methodologies for every skill: 5874

Ability to think practically and apply knowledge in practical situations was measured using the rubrics of the UPV for the assessment of the report of the group. The scale ranged from 1 (poor) to 4 (excellent). Ability to analyse and resolve problems was measured using the rubrics of the UPV for the assessment of the report of the group. It was also measured by individual observation carried out by the lecturer. In both cases, the scale ranged from 1 (poor) to 4 (excellent). Teamwork and leadership ability was measured using co-evaluation of the group members based on the rubrics of the UPV. It was also measured by individual observation carried out by the lecturer. In both cases, the scale ranged from 1 (poor) to 4 (excellent). Awareness of current business issues was measured using the rubrics of the UPV for the assessment of the report of the group. The scale ranged from 1 (poor) to 4 (excellent). It was also measured by self-evaluation carried out by the student. The scale ranged from 1 (strongly disagree) to 5 (strongly agree). The reports were assessed by two different lecturers to avoid personal opinions and to reduce differences in the final grades. Table 1 shows the number of students and number of written reports assessed by course. Table 1. Number of students and written reports assessed Courses Nº students Observation Nº students Self and coevaluation Nº written reports Tourism Marketing (TM) 48 35 13 Marketing Management (MM) 105 125 34 Agri-Food Marketing (AFM) 48 50 14 Business Management in Construction (BMC) 23 23 5 Tables 2, 3 and 4 summarise the profile of the sampled students. We performed a cross tabulation for some features of the students. As result, there are significant differences by course for the percentage of Erasmus Exchange Students and for the percentage of students who worked or had worked in the last two years. However, there are not significant differences by course for the interest in the course measured in percentage. Table 2. Percentage of Erasmus Exchange Students TM MM AFM BMC Mean Erasmus Exchange Students 0 25.00 14.81 13.04 18.05 Not Erasmus Exchange Students 100.00 75.00 85.19 86.96 81.95 χ2 with 3 degrees of freedom = 12.2089 (p=0.0067) Table 3. Percentage of students who worked or had worked in the last two years TM MM AFM BMC Mean Working 40.00 64.17 40.74 69.57 57.56 Not Working 60.00 35.83 59.26 30.43 42.44 χ2 with 3 degrees of freedom = 11.0458 (p=0.0115) 5875

Table 4. Interest in the course measured in percentage TM MM AFM BMC Mean Very uninteresting 0 0.83 0 0 0.50 Uninteresting 2.86 1.67 4.35 0 1.99 Indifferent 5.71 3.33 13.04 17.39 6.47 Interesting 82.86 60.83 65.22 60.87 65.17 Very Interesting 8.57 33.33 17.39 21.74 25.87 χ2 with 12 degrees of freedom = 19.1195 (p=0.0857) 3 RESULTS 3.1 Ability to think practically and apply knowledge in practical situations Table 5 shows the results of the assessment of this generic skill using the rubrics of the UPV. We obtain a grade per group depending on the level of the course. The items reflect the whole process to perform the skill. Students from Tourism Marketing obtained the highest mean score (3.35). Students from the Business Management in Construction course obtained the lowest mean score (0.55 points of difference). The final assessment of the skill for every student consists of the mean of the items considered. Table 5. Assessment of Ability to think practically and apply knowledge in practical situations Level Items TM MM AFM BMC Identifies specific targets following the instructions 3.38 Evaluates the quality of information provided for application 3.31 I Establishes the process to follow to obtain a practical solution from the information, resources and limitations Proposes solutions and / or concrete actions after analyzing the situation 3.38 3.31 Sets specific targets in relation to the situations that are presented 3.15 Obtains the necessary information to address situations 3.09 II Evaluates the quality of information available for application 2.97 Draws up a coherent plan to resolve situations that are presented 3.24 Proposes control indicators for monitoring the plan 2.53 Sets / agrees specific targets in relation to complex situations that are presented Gets / reaches consensus about the information necessary to address complex situations 3.07 2.80 3.14 2.80 III Evaluates the quality (relevance) of information available for application Prepares / agrees a coherent plan to resolve complex situations that are presented Proposes / reaches consensus about monitoring indicators to track the plan 3.50 2.80 3.21 2.80 3.21 2.80 Mean 3.35 3.00 3.23 2.80 5876

3.2 Ability to analyse and resolve problems Tables 6 and 7 show the results of the assessment of this skill depending on the methodology. Tourism Marketing is again the course with the highest mean score for the report (3.35). However, it is the course with the lowest mean score using observation (2.53). Students from Agri-Food Marketing obtained the lowest mean score for the report (3.04) while students from Marketing Management obtained the highest mean score for the observation (3.36). In observation, there is a difference of 0.83 points between the highest and lowest mean scores, while there is a difference of 0.31 points in the grades of the reports. The final assessment of the skill for every student consists of 50% from the report and 50% from the observation. As the source is the lecturer in both methodologies, they have the same weight. Table 6. Assessment of Ability to analyse and resolve problems analyzing the report Level Items TM MM AFM BMC I Defines the problem describing clearly and concisely the most important aspects Uses the indicated information sources and selects the correct data Employs a learned method for solving the problem proposed 3.38 3.31 3.31 Analyzes the consistency of the solution obtained 3.38 II Identifies a complex problem, decomposes this problem into simpler parts and argues it Contrasts the sources of information and handles rigorous data Uses the methodology from their experience in order to solve the problem in an efficient and justified way 3.32 2.85 3.35 Chooses an optimal solution justified by criteria 3.06 III Analyzes the causes and effects of problems from a global long-term approach Uses criteria of advanced search information for problem-solving and evaluates the quality of information Organizes systematically work for decision making (individual / group) Evaluates the possible solutions according to their scientific and technical feasibility and implementation difficulty 2.93 3.20 3.14 3.20 3.21 3.20 2.86 3.20 Mean 3.35 3.28 3.04 3.20 5877

Table 7. Assessment of Ability to analyse and resolve problems using observation Items TM MM AFM BMC The student asks meaningful questions about the proposed project. The student is able to argue the quality of its proposals when asking questions. 2.58 3.37 3.19 3.22 2.50 3.30 3.19 3.22 The student requests appropriate information. 2.50 3.42 3.27 3.22 Mean 2.53 3.36 3.22 3.22 3.3 Teamwork and leadership ability Tables 8 and 9 show the results of the assessment of the skill depending on the methodology. In coevaluation, there are not big differences between the highest (3.87 for Tourism Marketing) and lowest (3.66 for Agri-Food Marketing) mean scores. Nevertheless, in observation, there is a difference of 1.16 points between the highest (3.59 for Business Management in Construction) and the lowest (2.43 for Tourism Marketing) mean scores. The final assessment of the skill for every student consists of 20% from the co-evaluation and 80% from the observation. As the observation is carried out by the lecturer, it has a greater weight. Table 8. Assessment of Teamwork and leadership ability using co-evaluation Level Items TM MM AFM BMC Accepts and fulfills the team goals 3.90 I Attends and participates actively in the team meetings 3.73 Performs the tasks that are assigned within the team within the deadline set 3.97 Participates in the planning of the objectives 3.71 II Acts to deal with the conflicts of the team 3.72 He/she has committed to the realization of the collective tasks 3.72 III Contributes to the cohesion of the team in the way he/she communicates (collaborator role) Engages in the management and operation of the team (collaborator role) Proposes ambitious objectives and contributes to the distribution of tasks in a balanced manner (coordinator or leader role) Tracks and controls the tasks, deadlines and quality of the results (coordinator or leader role) 3.76 3.78 3.68 3.65 3.60 3.74 3.60 3.83 Mean 3.87 3.72 3.66 3.75 5878

Table 9. Assessment of Teamwork and leadership ability using observation Items TM MM AFM BMC The student performs the tasks that are assigned within the group within the prescribed period. The student participates actively in the meetings, sharing information, knowledge and experiences. The student contributes to the cohesion of the group in the way he/she communicates and interacts. The student contributes to the establishment and implementation of team work processes. - 3.49 3.23 3.48 2.53 3.45 3.19 3.78 2.50 3.44 3.08 3.65 2.47 3.35 3.08 3.70 The student conducts meetings effectively. 2.25 3.39 3.19 3.43 The student proposes to the group ambitious and clearly defined goals. The student facilitates positive management of differences, disagreements and conflicts that occur within the group. 2.14 3.40 3.19 3.57 2.69 3.49 3.27 3.52 Mean 2.43 3.43 3.18 3.59 3.4 Awareness of current business issues Tables 10 and 11 show the results of the assessment of this skill depending on the methodology. Analyzing the results of the report, students from Business Management in Construction obtained the highest mean score (3.40) and students from the Marketing Management course obtained the lowest mean score (3.11). Analyzing the results of the self-evaluation, students from Marketing Management obtained the highest mean score (3.27) and students from the Agri-Food Marketing course obtained the lowest mean score (2.82). It should be noted that the grades obtained through self-evaluation are lower than those obtained through the report. Marketing Management is the only course that does not meet this statement. Furthermore, the scale of the self-evaluation ranges from 1 (strongly disagree) to 5 (strongly agree) instead of 1 (poor) to 4 (excellent). This difference in the scale aggravates differences of the grades obtained between both methodologies. Obviously, it could be considered much better to use the same range for the scale in order to avoid these problems when comparing grades from different methodologies. The final assessment of the skill for every student consists of 80% from the report and 20% from the self-evaluation. As the report is assessed by the lecturer, it has a greater weight. Table 10. Assessment of Awareness of current business issues analyzing the report Level Items TM MM AFM BMC I Identifies which contents of the course are related to a contemporary problem Identifies / lists contemporary problems related to his/her professional field Describes the problem and its main features with appropriate vocabulary in different contexts orally or in writing Identifies valid sources of information related to the problem assigned (reliable sources, independent, recognized,...) Identifies / explains solutions to the problem based on the information available 3.08 3.38 3.31 3.46 3.38 5879

Identifies the causes that have led to the current situation problem Identifies the social, economic, cultural and other types of the problem implications in the context of his/her professional field 2.91 2.88 II III Distinguishes the parts of a problem and relates these parts, identifying the main actors in different dimensions (economic, social, ethical, technological...) 3.29 Reasons (criticizes) the proposed solutions to a problem 3.24 Proposes new solutions to the problem from the experience and information available Proposes solutions to certain important contemporary problems in his/her field and related fields Assesses the proposed solutions to major contemporary problems of his/her field and related fields Prioritizes the best solution to the problem from their own experience and information available 3.24 3.57 3.40 3.50 3.40 3.07 3.40 Reformulates the problem in terms of a new scenario 3.43 3.40 Evaluates the consequences and implications of the proposed solutions to the problem in terms of a new scenario 3.00 3.40 Mean 3.32 3.11 3.31 3.40 Table 11. Assessment of Awareness of current business issues using self-evaluation Items TM MM AFM BMC I knew the real conflict proposed by the company. 2.49 3.33 2.67 1.22 I have expanded the information I had about the current conflict. 3.31 3.42 2.78 3.65 I understood the real conflict from new perspectives. 3.29 3.37 3.12 3.65 I learned to face the real conflict of the company. 2.65 3.32 2.93 3.57 The solutions proposed by the group have helped me to deepen the current problems of the company. The project carried out has helped me to know the company. The project carried out has helped me to understand the problems that I will face in the future. 3.11 3.22 2.93 3.65 3.31 3.17 2.46 3.65 2.86 3.05 2.85 3.35 Mean 3.00 3.27 2.82 3.25 4 CONCLUSIONS The aim of this paper was to show how PBL can be used to evaluate students attainment of four generic skills. The four skills were: Ability to think practically and apply knowledge in practical situations, Ability to analyse and resolve problems, Teamwork and leadership ability and Awareness of current business issues. Different methodologies were used for the assessment of every one of them. When we combined different methodologies for assessing one generic skill, an agreement had to be reached for the weights of every methodology. We agreed to give more weight to the methodologies carried out by the lecturer. Students were very optimistic when assessing Teamwork and leadership ability through coevaluation. In fact, we can find the greatest grades of all the skills considered in this case. Instead, students were more realistic when assessing Awareness of current business issues through self- 5880

evaluation. The grades obtained by the students were lower than the grades obtained by the lecturer analyzing the report using the rubrics of the UPV. Analyzing the different methodologies, we realise that there are big differences by course in observation. Reports are evaluated by two lecturers to reduce subjectivity. However, observation is carried out by the lecturer who is in charge of the class, this is to say, only one. Maybe this is the reason why we find so great differences by course in the assessment of the skill when using observation. Although it is carried out by the lecturer who is in charge of the class, it requires a lot of work and time. As a result, all the students don t have a grade from observation. Other problem is that you never have an objective evidence of the assessment through this methodology. You only have your own opinion or your personal judgement in your head or memory. We could deduce that this is not an objective good method when assessing skills. The rubrics developed by our University were a very useful tool when determining the items to assess. As every item had a grade from 1 (poor) to 4 (excellent), you could calculate the mean or the average. It is always better to have an objective evidence to evaluate the skill. The reports and the questionnaires for the self and co-evaluation were proofs of that assessment. In this way, PBL can give to the lecturers a way to measure the development of these skills by the students. There are differences in the mean scores obtained by course but we didn t find differences by degree. In other words, there are not differences in the mean scores obtained by the bachelor s degree (Tourism Management or Business Administration and Management) and the mean scores obtained by the master s degree (Agricultural Engineering or Construction). We could deduce that there are not differences based on the student profile of every level. In short, the PBL provides enough indicators to assess the degree of achievement of these four generic skills and to determine whether students develop these skills to an acceptable level. REFERENCES [1] Fernández, I.; Castro, E.; Conesa, F. and Gutiérrez, A. (2000). Las relaciones universidadempresa: entre la transferencia de resultados y el aprendizaje regional. Revista Espacios 21(2), pp. 127-148. [2] Wood, D.F. (2003). Problem based learning. BMJ. 326, pp. 328-330. [3] Taboada, G.L.; Touriño, J. and Doallo, R. (2010). Innovación docente en el EEES de cara a la práctica profesional a través del aprendizaje basado en proyectos. XVI Jornadas de Enseñanza Universitaria de la Informática (JENUI). [4] Gülbahar, Y. and Tinmaz, H. (2006). Implementing project-based learning and e-portfolio assessment in an undergraduate course. Journal of Research on Technology in Education 38(3), pp. 309-327. [5] Engel, C.E. (1992). Problem-based learning. British Journal of Hospital Medicine 48(6), pp. 325-329. [6] Escribano, A. and Del Valle, A. (2008). El aprendizaje basado en problemas. Una propuesta metodológica en Educación Superior. Narcea Ediciones, Madrid. [7] Martí, J.A. (2010). Educación y Tecnologías. Servicio de Publicaciones de la Universidad de Cádiz, Cádiz. [8] Badía, A. and García, C. (2006). Incorporación de las TIC en la enseñanza y el aprendizaje basados en la elaboración colaborativa de proyectos. Revista de Universidad y Sociedad del Conocimiento 3(2), pp. 42-54. [9] Fernández, A. (2006). Metodologías activas para la formación de competencias. Educatio siglo XXI 24, pp. 35-56. [10] González, J. and Wagenaar, R. (2003). Tunning educational Structures in Europe. Informe final. Universidad de Deusto. 5881