AN EMPIRICAL APPROACH TO EVALUATE STUDENTS PARTICIPATION IN FREE/OPEN SOURCE SOFTWARE PROJECTS Sulayman K. Sowe, Ioannis Stamelos and Lefteris Angelis Department of Informatics, Aristotle University 54124 Thessaloniki, Greece Tel:+30-2310-998236 Fax:+30 2310998419 ABSTRACT Despite the popularity and global adoption of Free/Open Source Software (F/OSS), the implementation of teaching and learning methodologies in the context of traditional software engineering courses is still a novelty. This paper summarizes the approach we used to evaluate the participation of students in a F/OSS teaching and learning framework. The framework was implemented as a pilot study in which students volunteered to participate in software testing in F/OSS projects. The evaluation approach we discuss here is based on (a) students' participation in software testing and (b) results from two online surveys. In the evaluation process we utilized a statistical approach (both descriptive and inferential) we considered appropriate for assessing the students. We hope that our evaluation approach will help us further understand and strengthen empirical and experimental software engineering research in this emerging technology. KEYWORDS Teaching and learning, Software Engineering Education, Software Testing, Open Source Software Projects, Empirical Study. 1. INTRODUCTION As well as being a new style (Bazaar as opposed to Cathedral style [7]) of developing software aided by developments and advances in Internet technologies, Free and Open Source Software (F/OSS) is an emerging and fast evolving teaching and learning process. However, if this context is to survive the digital age and becomes ubiquitous in the future, there needs to be sustained efforts aimed at understanding and developing teaching and evaluation methodologies. There exists great number of empirical studies in F/OSS, but most of the research conducted in this domain has concentrated on the F/OSS development process and communities. The theory of learning underpinning F/OSS is one of constructivism in the sense that participants construct meaning by interacting with the environment in which the knowledge of the domain is not separated from the context in which it applies. F/OSS projects as bazaars of learning [9] offer a meaningful learning context in which students can be exposed to genuine software development activities. Valuing the learning benefits inherent in F/OSS projects and our long term involvement in empirical and experimental software engineering research in F/OSS, we were motivated to run a pilot study in order to provide opportunity for our students to work on what they considered interesting themselves and give them real-world experience in dealing with large projects [9]. The raison d'être for undertaking this pilot study was to (a) assess the feasibility of conducting a full-scale study with our students, (b) develop and test a research method, (c) identify any pitfalls which might occur using the identified research method, and (d) develop a an approach to evaluate students participation in F/OSS project. Because of the limited space, this paper discusses only the latter. 304
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2006) 2. RELATED WORK There is great emphasis on pragmatic teaching so that what is learnt benefit students long after graduation. The joint IEEE/ACM curriculum guidelines [4] suggest that CS curricular should incorporate Capstone projects which will give students significant experience in software projects. In this regard, many efforts have been made by involving students in software projects in local companies [1]. However, most companies are not willing to sacrifice their products to students and in most cases students have to complete their assigned projects in one semester. These debacles make it difficult for students to be involved in large and long-term projects [9]. There is increase interest in the F/OSS learning environment [8] and in F/OSS projects as bazaars of learning [9]. F/OSS is both an alternative teaching methodology and an educational model. Computer science students can be involved in meaningful software development activities [9] and get experience in dealing with realistic software systems with large quantities of code written by other people [3]. Many universities have also started teaching F/OSS course [5, 6 (pp.79-88)]. Projects (e.g. Edukalibre) have been lunched to study the transfer of F/OSS practices to the production of educational resources [2] and workshops (e.g. tossad) have started discussing the adoption of F/OSS in education [6]. The work we presented here is one way in which SE educators can transfer the F/OSS methodology to the educational domain. We discuss how to evaluate students participating in software testing in F/OSS projects. 3. BACKGROUND OF THE PILOT STUDY The pilot study was implemented in the Introduction to Software Engineering course (ISE) at the Department of Informatics, Aristotle University, Greece. The study lasted 12.5 weeks with 13 students and 2 lecturers. During the first phase students were introduced to the concept of F/OSS and guided to select projects from Sourceforge.net [10]. Students went through an exploratory session where they browsed through various projects. At the end of this phase, each student prepared a report on the F/OSS projects landscape, giving a brief history and characteristics of the project/s he would like to test in. During the second phase students selected and registered in their projects. Each student sent his/her project name and login details to the lecturer. These were used to track students activities in their projects. Students download the software to be tested and applied various software testing techniques such as smoke tests, functional tests (i.e. based on boundary values), usability tests, etc. Any discovered bug is logged into the project's bug database using standard bug reporting procedure and tools (e.g. Bug Tracking Systems). Where a student was not able to find a bug, he/she could run more tests on the software or select another project to continue testing. Every time a student finds or submitted a bug report, he/she notified the lecturer by email. At the end of this phase the students made another presentation detailing their experiences, stating the types of bugs found, how the bugs were found, what they thought caused the bugs, how they reported bugs, what responses, if any, were received from other participants, and any other problems they encountered. In the third phase students made their final fifteen minutes presentation in which they listed: particulars (email, student id), project details (name, login id, website, screenshots), likes and dislikes, future plans in the project/s, and list of testing activities (number of bugs found (bfn), bugs reported (brp), bugs fixed (bfx), and number of replies (rep) received). Students should give the URLs of the variables brp, bfx, and rep so that we can confirm their testing activities. 4. EVALUATION APPROACH At the end of the pilot study we evaluated the students based on (1) the presentations they made in class, (2) their email communications with the lecturer, (3) their testing activities. Their opinions and experiences in testing in F/OSS projects were captured in 2 surveys we conducted. Except during phase 1 and during class presentations (which also acted as brainstorming sessions), our only source of contact with the students was through email exchanges (135 emails in 87days). The evaluation approach we discuss in this paper ONLY cover the evaluation method 3 students testing activities and results of the 2 surveys. 305
4.1 Students Testing Activities In evaluating students testing activities, we considered the four variables (bfn, brp, bfx, and rep). 4.1.1 Descriptive statistics We used descriptive statistics as the basic feature to describe the data on students testing activities. When combined with simple graphics plots, descriptive statistics form the basis of virtually every quantitative analysis of data and we found this approach useful in providing us simple summaries about our data as shown in Table 1. Table 1. Descriptive statistics of students testing activities bfn brp bfx rep Number of students 13 13 13 13 Mean 5.54 5.23 1.15 3.31 Median 4.00 4.00 1.00 3.00 Mode 3.0 4.0 0 1 Std. Deviation 3.017 2.743 1.281 2.175 Sum 72 68 15 43 In total, 13 students tested in 16 F/OSS projects, found 72 bugs, reported 68, fixed 15, and received 43 replies from the F/OSS communities in their projects. The mean values of bugs found and reported per student were 5.54 and 5.23, respectively. These figures show that students reported slightly less bugs than they found, because some of the bugs they found were already reported. Even though students performed well in finding and reporting bugs in their projects, they did not do well in fixing bugs (mean=1.15). This is because students were not required to do any coding. 4.1.2 Bivariate Correlations Bivariate correlation analysis was performed on the variables. The Pearson correlation coefficient r was computed for all pairs of variables along with its significance p. Table 2. Bivariate correlation computed for the four variables Variables brp bfx rep bfn r.960.106.620 p.000.730.024 brp r -.035.490 p.911.089 bfx r.281 p.353 The results (Table 2) show a high correlation (r=0.960, p<0.001) between bfn and brp. The correlation between bfn and rep is also statistically significant (r=0.620, p=0.024). Since not all the bugs found were reported, the most interesting correlation is between brp and rep. This shows the actual interaction between the students and other project participants. The Pearson coefficient for this pair is 0.490 with p=0.089, indicating a rather moderate correlation. There was a negative correlation between brp and bfx (r=-0.035) which, however, is not statistically significant (p=0.911). 4.1.3 Factor analysis In order to investigate further the relation between the four variables we used factor analysis. Factor analysis is able to reveal latent variables or factors that are responsible for the correlation structure of a multivariate dataset. We used the Principal Components Analysis method for factor extraction and the Varimax rotation. The analysis showed that there exist two latent variables that can explain the behavior of the students. The first factor is highly correlated with bfn, brp and to some extend rep, while the other factor is clearly correlated with bfx. The relation between the original variables and the factors (component 1 and 2) extracted are shown in Figure 1(a). The factor scores of each student were estimated by the Anderson-Rubin method and they are plotted in Figure 1 (b). 306
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2006) Figure 1. Factor components (a) and Students factor scores (b) From the plot in Figure 1(b), we can see that student 2 has high scores in both factors and this shows an overall good performance. Student 6 has the highest score for factor 1 but low score for factor 2 and this shows a good performance in the three variables (bfn, brp, rep) associated with factor 1. Students 3, 5, 8, 9, 10 and 12 are clustered around the same region. This group has a low to moderate performance in all of their activities. Another interesting grouping is formed by two students (4 and 11) who although they did not perform so well in the variables associated with the factor 1; they have good performance for bfx which is associated with factor 2. 4.2 Students Experiences and Opinions: Surveys We invited the students via email to complete two anonymous online surveys 1. The surveys items were meant to measure the pedagogical value of the study; students' opinions and experiences about software testing in F/OSS projects, and how much they have benefited by contributing and learning from testing in F/OSS projects. Survey 1 was conducted in week 6 and consisted of 21 items. We called it an intervention survey because it allowed us to intervene early and focus attention on difficulties students were having (e.g. ease of finding a project, process of reporting bugs). Survey 2 was conducted in week 13 and consisted of 19 items. The response rate for both surveys was 84.62% (11 out of 13 students). Seven items (Table 3) from survey 1 were repeated in survey 2 so that we could compare students' responses to the common questions and see how their motivation and perception has changed overtime. Table 3. Questions common to both surveys Survey 1 Survey 2 Items/Questions Variables Do you enjoy software testing in F/OSS projects? Q1 Q3 Did you find it easy to get a project to participate in? Q2 Q18 Was it easy to find bugs in the software in your project? Q3 Q5 Was the process of reporting bugs easy? Q4 Q6 Would you have preferred to do other courses in Open Source Software? Q9 Q8 Did you read and understand the bugs others reported in your project? Q10 Q11 Are you considering participating in the project you selected after you graduate? Q20 Q14 For all comparisons, the McNemar test for two related dichotomous variables was used. This test is especially useful for detecting changes in responses between surveys. For every comparison we give the significance of the test (p). From the McNemar test we were able to make the following conclusions: I. ST1Q1 vs ST2Q3: The test revealed no comparison. All 11 students responded "Yes" in both surveys. This means that throughout the pilot study students enjoyed software testing in their projects. 1 http://swserv1.csd.auth.gr/survey/index.php?sid=15 and http://swserv1.csd.auth.gr/survey/index.php?sid=18 307
II.ST1Q2 vs ST2Q18: In both surveys the answers are the same. There is no statistically significant difference (p=0.549). The students expressed that it was not easy to find a project to participate in. III. ST1Q3 vs ST2Q5: The answers are almost the same. There is no statistically significant difference (p=0.754). Most students found it difficult to find bugs in their software. IV.ST1Q4 vs ST2Q6: There seems to be a statistically significant difference (p=0.039). Even though finding bugs was difficult, most of the students reported the bugs they found with relative ease. However, this was easy at the beginning but became gradually difficult as students could not find new bugs. V.ST1Q9 vs ST2Q8: There is a statistically significant difference (p=0.002). This shows that most of the students would prefer to have their other courses taught using F/OSS methodology. VI.ST1Q10 vs ST2Q11: There is some indication of statistically significant difference (p=0.065). Students expressed that they read and understood bugs others reported in their projects. VII.ST1Q20 vs ST2Q14: There is a statistically significant difference (p=0.008). Towards the end of the pilot study all of the students were considering participating in their projects after they graduate. 5. CONCLUSION In this paper we have presented an approach we used to evaluate students involved in software testing in F/OSS projects. The correlations show how significant one variable is compared to others. Factor analysis was used to reveal latent factors responsible for the correlations in the multivariate dataset. In the surveys McNemar test was used to detect changes in students' responses between items. This evaluation approach is important in the sense that it can be extended and used in a similar study involving larger sample of students. Its strength lies in its ability to reduce the dimensionality of the data and reveal groupings of students' behaviours. Compared to formally structured SE courses in most universities, teaching and learning from F/OSS projects has some peculiar characteristic. For example there is no formal curricular or timetabling structures, students can do their work (software testing) whenever and wherever they want with little or no guidance from the lecturer. Evaluating students performance in such an environment needs further research by SE educators and curriculum designers and implementers. In light of this, our evaluation approach serves as a significant yardstick for future research. REFERENCES 1. Alzamil, Z. 2005. Towards an effective software engineering course project. In Proceedings of the 27th International Conference on Software Engineering, ACM Press, pp. 631-632. 2. Barahona, J. M., Tebb, C., Dimitrova, V. 2005. Transferring Libre Software development Practices to the Production of Educational Resources: the Edukalibre Project. In First International Conference on Open Source Systems, Genova, Italy. 3. Carrington, D., Kim, S. 2003. Teaching Software Engineering Design with Open Source Software, 33rd ASEE/IEEE Frontiers in Education Conference, Nov. 5-8, Boulder, CO. 4. IEEE/ACM Joint Task Force on Computing Curricula, Retrieved November 27, 2005, from http://sites.computer.org/ccse/se2004volume.pdf 5. German, M. D. 2005. Experience teaching a graduate course in Open Source Software Engineering. In Proceedings of the first International Conference on Open Source Systems, Genova, pp.326-328. 6. Ozel, B., Cilingir, B., Erkan, K. 2006 (Eds.) Towards Open Source Software Adoption. OSS 2006 tossad workshop proceedings, Como, Italy 7. Raymond, S. E. 1999. The Cathedral and the Bazaar, O'Reilly, Sebastopol, USA. 8. Sowe, S.K., Karoulis, A., Stamelos, I., Bleris. G.L. 2004. Free/Open Source Software Learning Community and Web-Based Technologies, IEEE Learning Technology Newsletter,Vol.6(1), 2004. pp 26-29. 9. Sowe, S. K., Stamelos, I., Deligiannis, I., 2006. A Framework for Teaching Software Testing Using F/OSS Methodology. In IFIP International Federation for Information Processing, Open Source Systems, Vol. 203, pp. 261-266. 10. Sourceforge.net. Available at: http://sourceforge.net/ 308