Orit Hazzan's Column Qualitative Research in Software Engineering Orit Hazzan Department of Education in Technology and Science Technion, Haifa 32000, Israel oritha@tx.technion.ac.il This column is coauthored with Yael Dubinsky Department of Computer Science, Technion, Haifa 32000, Israel Introduction This column discusses the application of qualitative research for research in Software Engineering (SE). Specifically, we outline what research questions are suitable for investigation using qualitative research, what research tools are used for data gathering in qualitative research and how the qualitative data analysis process guides the formulation of answers to the research questions. Accordingly, the column provides basic knowledge for determining whether qualitative research is suitable for the investigation of specific research questions in SE. By doing so, the column contributes to researchers in the field and assists them in deciding whether to adopt the qualitative research approach. The importance of the column stems from the fact that only very few works presented in the SE literature contain some qualitative experimental component. For example, according to Glass, Ramesh and Vessey (1994), SE research produces products that are almost exclusively technical. Specifically, SE research examines topics related to systems and software concepts at technical levels of analysis by formulating processes, methods and algorithms. Furthermore, according to Glass et al., most SE research studies (58%) use formulative approaches, e.g., formulation of an algorithm, some studies (28%) use descriptive research approaches, while even fewer are evaluative in nature (14%). This column is based on our experience with qualitative research processes in a variety of SE situations. Accordingly, the illustrative examples, presented in this column to demonstrate the suitability of qualitative research for research in SE, are taken from our database.
Background The qualitative research approach is usually used for the investigation of social phenomena, or in other words, situations in which people are involved and different kinds of processes take place. Within this arena, qualitative research is usually conducted in cases in which what we wish to learn about environments, situations and processes cannot be retrieved by quantitative data analysis methods. Indeed, quantitative data analysis can shed light on many aspects of such situations and may enable us to argue with a certain degree of generalization. The nature of quantitative research does not, however, enable the researcher to explore complex situations in depth. By the same token, we are not claiming that the qualitative research approach enables us to present a full picture of complex situations. We are proposing, however, that the qualitative research approach enables us to highlight many angles of people-centered situations. This perspective is illustrated in Table 1. Table 1: Example of a situation in which qualitative research enables to highlight a peoplecentered perspective One situation in which we adopt qualitative research approach is in learning about software organizations prior to the assimilation of a new software development method. In most cases, quantitative data about the software development processes in the said organization is available to those in the organization who initiate the process; they wish, however, to learn also about the practitioners' perspective on the current situation, as well as the practitioners' willingness to make the transition to a new development process. For this purpose, we employ the qualitative research approach and conduct a qualitative-based organizational survey prior to such assimilation processes. In such qualitative-based organizational surveys we mainly address human aspects related to software development processes. The qualitative-based organizational surveys are based on observations at the organization sites, interviews with different role holders in the organization, questionnaires that are completed by different role holders in the organization and meetings with different role holders in the organization management. Since we use a variety of data-gathering tools, when data is analyzed, the findings can be validated by illustrating how the same finding is supported by different data items. In most cases, the themes that emerged from the data analysis address human-related factors in SE, such as the customers, software development culture, and teamwork. The added value of such an approach is that the practitioners' voice is heard and, based on these voices, guidelines for the assimilation of the new software development method are outlined. Qualitative Research - Essence, Connections to Other Research Approaches and Application The main characteristic of qualitative research is that instead of aiming at accepting or rejecting an a priori defined hypothesis, research works that use a qualitative approach aim at constructing a theoretical framework that emerges from the analysis of the data gathered during the research and enables to explain the research results in a coherent manner. Such a framework is called a
grounded theory (Glaser and Strauss, 1967). Glaser and Strauss explain: Generating a theory from data means that most hypotheses and concepts not only come from the data, but are systematically worked out in relation to the data during the course of the research. (p. 6). This approach inspires a special research setting and process. Specifically, in order to construct a grounded theory, qualitative research is characterized by a spiral structure, in which each phase is based on the previous stages and elaborates on the framework that has been constructed so far. Metaphorically, the qualitative research approach can be described as an on-going dialog between the researcher and the research field, through which the former improves his or her understanding of the latter. This approach, which intertwines the development of a theory together with the research process itself, is therefore especially suitable for the investigation of topics that have not been previously researched. Table 2 presents an example of a research work in which such a process was applied. Table 2: Example of a grounded theory construction process This example deals with teaching software development methods in a project-based software development capstone course (Dubinsky, 2005; Dubinsky and Hazzan, 2005). The goal of the research was to construct a teaching framework for software development methods in higher education. For this purpose, the teaching of software development methods was examined in a project-based course in which Extreme Programming was used. To achieve the research target, Action Research (Lewin 1946, reproduced in Lewin 1948) was conducted, in which the teaching framework was constructed iteratively and each research cycle was composed of planning action collection of data available as a result of the action data analysis reflection refinement of the constructed framework. Specifically, the academic environment enabled the establishment of an iterative process, during which the proposed teaching framework was shaped through an inductive process, while the results obtained were constantly applied, and the suitability of the developed framework was continually examined and compared to newly gathered data and findings through the well-defined constant comparison process. As the research progressed and the teaching framework started to be shaped and refined, the question of when to stop the research cycles was considered. Specifically, questions such as the following were raised: How does one know how many research cycles are sufficient? Attempts to answer such questions clarified that no final version of the teaching framework can be formulated and that it should have a dynamic nature. In other words, it was clear that such a teaching framework can not be static; changes in technology, in SE methods and in teaching and learning approaches will continue to affect the proposed teaching framework. In practice, the dynamic nature of the teaching framework will provide future teachers using it with the ability to refine and maintain it according to their needs and those of their students. An examination of the previous paragraph reveals that questions of this kind also emerge with respect to software development processes, for instance: How does one know when the software is ready to be delivered to the customer and shipped to end users? In fact, it is a well-known fact that the life cycle of software does not end when it is shipped to the users, and that the maintenance phase enables users to introduce and ask for changes in the software tool according to their needs.
This similarity between the two processes led, at the first stage, to the formulation of an analogy between action research and an agile software development method. In a later phase, it turned out, that this analogy is also suitable as a basis for the process of teaching software development methods in a computer science project-based capstone course. In fact, analysis of the emerged teaching framework revealed that it fits and extends the above analogy. Thus, the theory that emerged from the research consisted of a 3-dimensional analogy between Action Research, Agile Software Development Methods, and Teaching Software Development Methods. The construction of a grounded theory is a meta-characteristic of the qualitative research approach from which stem most of its other characteristics. Due to space limitations, we mention in what follows only three additional important characteristics of the qualitative research approach. First, the data and products of qualitative research are verbal. With respect to the data collection tools, this characteristic of the qualitative research is expressed by the fact that the main data gathering tools are interviews and observations. Second, the data analysis methods employed in qualitative research aim at directing the researchers to interpret the data from the perspective of the participants in the investigated situation, i.e., to understand the meaning that the participants in the research associate to the researched phenomenon. Third, with respect to the question of the generalization of qualitative research, we claim that generalization has a different meaning in the case of qualitative research. As mentioned before, the main target of qualitative research is to construct a grounded theory. In order to do so, the course of a typical qualitative research work is iterative, and is based largely on data collection by means of interviews and observations. As a result of such a research setting, a qualitative research work usually focuses on a relatively small number of participants who are part of the research field. In order to enable the potential reader of a qualitative research to evaluate the relevance of the research findings to the case he or she is dealing with (in other words, to assess the level of generalization of the results of a qualitative research), the research participants are selected very carefully and the description of the research field and its results is very detailed (this writing style is called "thick description"). Table 3 presents an example that illustrates this perspective. Table 3: Example of a study that illustrates generalization in qualitative research In Dubinsky and Hazzan (2003) we presented a framework for coaching student projects in computer science capstone courses. This research was based on a retrospective process of four coaches who coached and guided students in the development of software projects within the framework of Extreme Programming. The retrospective data were collected using qualitative research tools that included open questionnaires and a series of interviews. The analysis of this gathered data was aimed at eliciting the meaning attributed by the coaches to a new situation, whereby they were required to guide their students in software development processes.
The analysis of the retrospective data yielded six categories that formed the framework for coaching student software projects. The Project category emphasizes the management of resources needed for the project, such as a time schedule and various organizational aspects. The Method category addresses the software development method s practices and the tools used in the project. The Development Team category focuses on the development environment and on communication among team members. The Customer category introduces the business aspect into the project and focuses on requirements and product acceptance. The Feelings category refers to the people involved in the project, from the viewpoint of their inner being. The last category, Coaching Team, emphasizes the support given to the coaching team in order to maintain continuous learning and receipt of feedback. Naturally, there are overlaps between these six categories; from the analysis of the retrospective data, however, it is seen that each of these categories plays a significant role. The detailed description of the research presented in Dubinsky and Hazzan enables readers to judge the degree of generalization of the research results with respect to specific situations they are dealing with. Relationships between qualitative and quantitative research approaches This sub-section explores the combination of quantitative and qualitative research approaches. One option for such a combination is to start with a qualitative research work, trying to identify the important observations as they are revealed by the participants in the research field. Then, based on the findings of the first stage, several hypotheses are tested in a quantitative manner and then, based on the findings of the second phase, a second qualitative research phase is performed, that aims at explaining those quantitative findings. Table 4 presents another way in which qualitative and quantitative approaches may be combined in a single research study. Table 4: Example of a study that illustrates the combination of qualitative and quantitative research approaches The research described by Dubinsky, Hazzan, Talby and Keren (2006) examined the transition to an agile development process in a large-scale software project in the Israeli Air Force, as it was perceived from the system analysis and design perspectives. Specifically, during the first half-year of transition, the project specifications of an agile team are compared with those of a team that continued to work according to the previous "heavyweight" method. Size and complexity measures were used as the basis of the comparison. In addition to inspecting the specifications, the change in the role of the system analysts, as conceived by the system analysts, was examined. Using a quantitative research approach, specifications produced from both kinds of teams the traditional one and the agile one were examined and compared. From a qualitative research approach, we wished to understand the process from the system analysts and designers' point of view. Accordingly, we interviewed system analysts and asked them questions such as Do you feel that your role has been changed? If no, please describe your role before and after the transition. If yes, please describe how your role has been changed., Please compare the traditional way with the agile one., and so forth. The combination of the quantitative and the qualitative research approaches enabled to present a wider, as well as deeper, picture of how the transition process looks from the system analysis and design perspective.
Application of qualitative research So far, we discussed the usefulness of the qualitative research approach and what it can achieve. We also addressed some relationships between qualitative research and quantitative research. In what follows, we address datagathering tools and data analysis tools. We start with data gathering tools. The most common data gathering tools used in qualitative research are interviews and observations (which sometimes are recorded on videotape to enable repeated viewing). For example, one of the main research tools used in the research described in Table 2, which deals with teaching framework for software development methods, was observations of software development teams. The observations provided an opportunity to document the actions, behavior, reactions and additional environmental characteristics in the researched environment. In addition, open interviews with students and coaches were conducted, in which the participants in the research field were asked to describe their perspective of the teaching environment. The interviews with the students focused also on specific practices of the software development method. In addition to observations and interviews, other qualitative research tools exist for data gathering, such as researcher diary, reflections, questionnaires, artifacts and documents. For example, in the above research work on teaching software development methods, forum messages were also used as means for data gathering. In general, each data gathering tool can complement, deepen and broaden findings obtained using other data gathering tools. Table 5 presents an example of research that uses open questionnaires to elicit reflection processes. Table 5: Example of a study that illustrates the use of questionnaires to elicit reflection processes The research described by Talby, Hazzan, Dubinsky and Keren (2006) analyzed reflections of an agile team working on the development of a large-scale project in an industrial setting. The team used an Iteration Summary Meeting practice, that included reflection as one of its four elements. The technique used for the entire meeting, and for the reflection element in particular, is described in the column, and empirical evidence is given to show that the reflection was assessed as highly effective, achieving its intended goals, and increasing team satisfaction. The main research method used in this research was personal reflection of the team members on the reflection process, collected by means of written questionnaires completed several months after the reflections in question took place. Different data analysis methods also exist. The main data analysis method used in the construction of a grounded theory is inductive analysis. Inductive analysis is supported by the constant comparison technique, which guides the researcher to keep examining his or her findings, relative to information
constantly obtained from different sources (interviews, observations, etc.) and from different informants, at the different research stages. The research described in Table 2, for example, employs this approach extensively. Embracing qualitative research in SE We now explain why, in our opinion, qualitative research has not yet been largely embraced by the SE research community. Before we delve into the explanation, we would like to re-mention that qualitative research approach is used in the SE. For example, Sharp, Hovenden and Woodman (2005) report on using metaphor in a SE qualitative research, aiming to uncover non-technical factors affecting the adoption and evolution of software quality management systems. We propose, however, that the potential contribution of the qualitative research approach to SE research has not yet been fully exploited. We propose that the reason that the SE community does not use qualitative research more extensively is related to the communities within which SE research is conducted. Naturally, in most cases, SE research is conducted in Computer Science or SE departments, which, as mentioned earlier, do not commonly apply a qualitative research approach. Conclusion We summarize the column by presenting two main benefits that may be gained by using the qualitative research approach in SE: expanding the research scope and deepening specific research findings. First, qualitative exploration may enable us to expand our scope of research. The open nature of the qualitative research may lead us to new, and sometimes even unpredicted, research directions that were not considered at the onset of the research. Second, the qualitative approach may enable us to deepen our findings. As previously mentioned, in many cases, SE research addresses topics that deal with human-related processes. Such processes, by nature, are rich, consisting of many details and perspectives. Accordingly, it is reasonable to assume that if we approach these processes with a qualitative approach, which concentrates on the details that constitute them, we may deepen our understanding of such processes. In summary, and as mentioned before, we note that there is no one approach (quantitative or qualitative) that is preferable over the other. Yet some phenomena are more suitable for investigation using a qualitative research approach.
References Dubinsky, Y. and Hazzan, O. (2003). Extreme Programming as a framework for student-project coaching in computer science capstone courses, Proceedings of the IEEE International Conference on Software Science, Technology & Engineering, Herzelia, Israel, pp. 53-59. Dubinsky, Y. and Hazzan, O. (2005). A framework for teaching software development methods, Computer Science Education 15(4), pp. 275-296. Dubinksy, Y. (2005). Teaching Software Development Methods, Ph.D. Research Thesis, Technion Israel Institute of Technology. Dubinsky, Y., Hazzan, O., Talby, D. and Keren, A. (2006). System analysis and design in a large-scale software project: The case of transition to agile development, Proceedings of the 8th International Conference on Enterprise Information Systems, Paphos, Cyprus. Glass, R. L., Ramesh, V. and Vessey, I. (1994). An analysis of research in Computing disciplines, Communications of the ACM 47(6), pp. 89-94. Glaser, B. and Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research, Chicago, Aldine. Lewin, K. (1948). Resolving Social Conflicts; Selected Columns on Group Dynamics. Gertrude W. Lewin (ed.). New York: Harper & Row. Sharp, H., Hovenden, F. and Woodman, M. (2005). Using metaphor to analyse qualitative data: Vulcans and humans in software development, Empirical SE 10(3), pp. 343-365. Talby, D., Hazzan, O., Dubinsky, Y. and Keren, A. (2006). Reflections on reflection in agile software development, Proceedings of the Agile 2005 Conference, Minneapolis, Minnesota, USA, pp. 100-110.