Pair Programming: A Contingency Approach

Transcription

1 Pair Programming: A Contingency Approach Pair Programming: A Contingency Approach Abstract Carolina Salge University of Georgia csalge@uga.edu Research-in-Progress Nicholas Berente University of Georgia berente@uga.edu Pair programming (PP) is a practice when two developers work together on the same task using one computer screen (Beck, 2000; Williams & Kessler, 2003). The pair consists of a driver and a navigator. The driver s goal is to write the program code while the navigator s goal is to perform more strategic tasks such as reviewing the code and searching for errors. PP has drawn the attention of many researchers and practitioners because many believe it is a better alternative to individual programming. Yet, the current literature possesses conflicting results, which reflect a lack of consensus. In this paper we look to take a step toward reconciling these ambiguous findings by carefully distinguishing between dependent variables used in the various extant studies, and by developing a contingency approach to understanding the merits of PP. Keywords Pair programming, learning, quality, pair skill similarity, and task complexity. Introduction Pair programming (PP) is a practice when two developers work together on the same task using one computer screen (Beck, 2000; Williams & Kessler, 2003). The pair is composed of a driver and a navigator. The driver writes the program code and the navigator performs more strategic tasks including, but not limited to, reviewing the code, searching for errors, thinking of the overall structure of the code, finding better alternatives, and etc. The two developers switch their roles periodically or as needed. PP has drawn the attention of many researchers and practitioners because many believe it is a better alternative to individual programming. Several controlled experiments have concluded that PP is superior to individual programming. These studies assert that PP is better because (a) it reduces defect rates and the time required to solve the tasks correctly, (b) it improves design quality, and (c) it increases productivity, team learning, and job satisfaction (Williams, Kessler, Cunningham, & Jeffries, 2000; Vanhanen & Lassenius, 2007; Canfora, Cimitile, Garcia, Piattini, & Visaggio 2007; Heiberg, Puus, Salumaa, & Seeba, 2003). For example, Cockburn and Williams (2001) suggest that for a cost of about 15% of development time, PP improves design quality, reduces defects and staffing risk, enhances technical skills, improves team communications and is considered more enjoyable at statistically significant levels. In one recent study, PP has been found to require 29% less total time to complete a given task than individual programming (Venkata et al 2012). Other studies, however, have findings that contradict this positive view of PP. Numerous studies have found that PP does not reduce the time required to solve the tasks correctly and does not increase the proportion of correct solutions (Arisholm, Gallis, Dyba, Sjoberg, 2007; Nawrocki & Wojciechowski, 2001). Thus, there is a general lack of consensus on the merits (or lack therof) of PP, and the conditions under which PP can be more or less effective. In this paper we take a step toward reconciling these ambiguous findings by carefully distinguishing between dependent variables used in the various extant studies, and by developing a contingency approach to understanding the merits of PP. In doing so, we identify three potential outcomes (dependent variables) associated with pair programming: quality; effort; and learning. We further identify three contingency factors that moderate the effects of PP on these outcomes. These contingency factors are (1) pair skill level similarity; (2) task complexity; and (3) pair compatibility. The main goal of this paper is to investigate when is PP likely to be a more (or less) effective practice than individual Twentieth Americas Conference on Information Systems, Savannah,

2 SIGADIT programming? Our study contributes to the literature by conceptually addressing the current lack of consensus in the PP literature. We reconcile this lack of consensus by reviewing the literature and considering when PP is likely to be more (or less) effective than individual programming. The remainder of the paper is organized as follows. First, we provide a overview of research on PP and software development performance. Drawing on this review, we next develop a set of propositions explaining when PP practices are more (or less) effective than solo programming. We then conclude and suggest further research. Literature Review Pair Programming (PP) Software developers and researchers alike have increasingly become interested in PP because of its possible benefits. Sun (2012) found that most of studies that have investigated the effects of PP (compared to individual programming 1 ) were not necessarily the most rigorous studies - they involved either anecdotes from practice and were mainly published in conference proceedings. Among the more rigorous studies that found their way into refereed academic journal publications, nearly half focused on university computer science education rather than real world industrial practice. Sun further asserted that the academic and empirically focused portion of the literature is weak with respect to theory and conceptual foundations. In order to take a step toward providing a theoretical foundation for PP research, we started with a review the PP literature consistent with the recommendations from Webster and Watson (2001). Table 1 provides an example of this analysis for five recent articles (we did not include the entire analysis due to space considerations, but the Appendix lists the articles we used in our analysis). This is followed by a summary of two of these papers to provide a flavor for the research in this area: Venkata et al. (2012) and Salleh, Mendes, & Grundy (2011). Study Di Bella et al. (2013) Venkata al. (2012) et Salleh et al. (2011) Han et al. (2010) Research Method Sample Size Research Question / Purpose Field Case 34 Under which circumstances has the impact of PP on code quality been investigated? What are the main findings of the impact of PP on code quality? Mixed (Interview & Experiments) Systematic Literature Review and Meta- Analysis 55 How the effectiveness of pair programming increases by pairing students of different education levels? What are the benefits and challenges of pairing students? 74 studies (SLR) and 6 studies (MA) What evidence is there of PP studies conducted in higher education settings that investigated PP s effectiveness and/or pair compatibility for CS/SE education? Experiment 115 To analyze the educational effects of a peer-learning agent based on pair programming in programming courses Summary of Results PP provides a perceivable but small effect on the reduction of defects in these settings PP is more effective than individual programming in terms of defect reduction, defect density, person hours, and number of tests passed Students skill level was the factor that affected PP s effectiveness the most. Students satisfaction when using PP was overall higher than when working solo. The exchange of roles and the meaningful feedback between the peer-learning agent and the student have a positive effect on learning 1 Please see Table 2.1 in Sun (2012) 2 Twentieth Americas Conference on Information Systems, Savannah, 2014

3 Pair Programming: A Contingency Approach Hannay al. (2010) et Quantitative (EFA and CFA) 196 (98 pairs) To analyze the impact of the Big Five on the performance of pair programmers together with the impact of expertise and task complexity Personality traits, in general, have modest predictive value on PP performance compared with expertise, task complexity, and country Table 1. Literature Review (most recent 5 studies) Venkata et al. (2012), investigate two research questions: (1) How the effectiveness of pair programming increases by pairing students of different education levels? (2) What are the benefits and challenges of pairing students? They follow a mixed-method approach and use interviews to answer the first question and two experiments to answer the second question. Interview results suggest that professors recognize several benefits associated with PP, including increases in code productivity, communication skill, new coding idioms and quality. Professors further suggest that PP is beneficial to knowledge transfer, problem solving skill, and work satisfaction PP was also perceived to build trust and increase teamwork and student academic performance. Four PP challenges were then recognized: deciding on pairs, assigning the complexity tasks, and bonding between team members. Experiment one results show that PP is more effective than solo programming because code length, defects, and defect density were all significantly smaller for pairs when compared to individuals. In experiment two, quality and effort are considered so the authors can examine PP effectiveness of students from same level to the students from different levels (PhD, post-graduate, and undergraduate). The results from experiment 2 show that paired PhD students are the most effective followed by post-graduate and undergraduate, respectively PhD pairs passed more test cases, spent less time completing the tasks, and had fewer mistakes. Moreover, a PhD student, when paired with a post-graduate student performed best in the same criteria when compared to other combinations (post-graduate undergraduate pair, and PhD undergraduate pair). The second example study, Salleh, Mendes and Grundy (2011), investigates the effectiveness of PP as a pedagogical tool in higher education computer science courses. They performed a systematic literature review (SLR) of empirical studies and found that several compatibility factors can influence PP effectiveness. Skill level, however, was the most influential factor. Time spent on programming was found to be the most common measure used to gauge PP s effectiveness. Additionally, Salleh et al. (2011) find that students satisfaction when using PP was overall higher than when working solo. Their meta-analyses show that PP was effective in improving students grades on assignments. Their results further show a clear gap in the literature: few studies pay attention to pair compatibility as a possible factor influencing PP effectiveness. These examples illustrate themes that run through the broader literature on PP intensive use of students, multiple dependent variables, and different motivations for the studies. From this literature, we identified three broad dependent variables that can be characterized in terms of quality, effort, and learning. We chose these three constructs for effectiveness because they are the most commonly used in the literature 2. We do not claim this list is exhaustive, but these categories accommodate virtually all dependent variables in our review. Next we will briefly review each in turn. Quality. Several studies examine PP effectiveness in terms of software quality. Overall, our literature review suggests that there is little consensus on a single quality measure. Most studies tend to heavily focus on defect rates (Vanhanen and Lassenius, 2005), correctness (Lui and Chan, 2003), percentage of test cases passed (Williams et al., 2000), and classroom grades (McDowell et al., 2003) as quality factors. The majority of studies conducted in educational settings find a positive relationship between PP practices and software quality (Declue, 2003; Sanders, 2001; Cockburn and Williams, 2001; McDowell, Werner, Bullock, and Fernald, 2002; Williams et al., 2000). There are, however, studies finding mixed results for the improved quality assertion. Abrahamsson (2005), for example, finds no significant difference between PP and individual programming in terms of defect density. Two other studies show that PP initially predicted higher quality; pairs originally performed better than individual programmers in the first assignment. However, PP was no longer associated with higher quality in the next assignment. In fact, PP 2 Project duration is not included in our list because it can be derived from effort and therefore does not need to be a stand-alone dependent variable. Twentieth Americas Conference on Information Systems, Savannah,

4 SIGADIT performance was no different than solo performance in the second assignment (Hanks et al., 2004; Gehringer, 2003). Existing literature therefore shows that the effect of PP on quality is mixed. Effort. Most studies that examine the influence of PP on effort find that pairs spend more time (hours) in developing software/performing tasks than solo programmers. In fact, prior research shows that the average completion time for pairs is about 40% more than for individuals (Nosek, 1998). More recent research shows an even more drastic effect instead of spending 40% more time on tasks, pairs seem to spend double of the time (Rosta 11 her & Kericko, 2002, Nawrocki & Wojciechowski, 2001; McDowell et al., 2003). There are, however, findings contradicting these results. Lui and Chan (2003), for example, find that if we take into consideration the same amount of quality, pairs actually spend about 4% less time than individuals on the same task. Once again, we find inconsistency in the current literature. Learning. The majority of studies suggest that PP has a positive effect on learning. Amber (2007), for example, reports that PP facilitates the spread of knowledge and skills throughout teams helping new employees learn the environment and build bonds with other team members rather fast. Numerous studies suggest that PP, when compared to individual programming, transfers knowledge at a faster rate and therefore showing a positive impact on learning (Williams, 1999; Sanders, 2001; Rumpe & Schroer, 2002; Tessem, 2003; Vanhanen & Lassenius, 2005 and 2007). Cliburn (2003), however, reports a survey study indicating that several students claimed to learn more when they work by themselves. Hanks et al. (2004) do not confirm Cliburn s (2003) findings but also do not support the assertion that pairs learn more than individuals. Despite the two opposing studies, most of the literature suggests that PP is positively associated with learning. Literature Review Summary. The literature review we conducted for this study suggests that the current PP literature possesses a rather weak theoretical foundation and little consensus in empirical results. The majority of studies did not use theory to hypothesize proposed relationships. In fact, only three studies in our literature review propose a theoretical framework to analyze the effects of PP (Arisholm et al., 2007; Balijepally et al., 2009; Sun, 2012). The mixed results from the literature point to possible contingency variables that might be used in theory development into the effects of pair programming. Hypotheses Development In this section, we build on the findings of the literature as well as our own reasoning to assert three propositions. These are illustrated in Figures 1, 2, and 3. Note that our independent variable, across all models, is pair programming method versus solo programming. Next we address three main dependent variables: quality, effort, and learning. Quality: The Moderating Effect of Pair Skill Similarity Numerous articles assert that PP effectiveness depends on pair skill similarity. We define pair skill similarity as the extent to which pairs match their own programming qualifications and experiences. We assume that the solo programmer is at the same skill level as the better programmer in the pair. This assumption is needed because if the solo programmer is extremely skilled and the pair is not, then clearly the solo programmer will produce a higher quality-coding outcome. Two studies propose that PP is more effective than solo programming in reducing coding defect rates when pairs possess slightly different skill levels (Jensen, 2003; Van Toll III et al., 2007). A more recent study, however, contradicts these findings. Salleh et al. (2011) find that PP works best when the pair has a similar skill level. It is important to note that extreme skill differences in a PP context have not been found to improve quality. The primary advantages of PP over solo programming for improving quality are that PP enables continual design and code review. Navigators can call out every formatting error and syntax mistake done by the driver. As a result coding standards are followed more accurately given peer pressure. Navigators, however, will only be able to pressure drivers if they are, at least, as skilled as drivers. If the driver is much more skilled than the navigator then coding review is practically useless because the navigator won t be able to pinpoint the driver s errors. In fact, the driver might feel like he/she is doing most of the work and therefore may not be satisfied with PP. Therefore: 4 Twentieth Americas Conference on Information Systems, Savannah, 2014

5 Pair Programming: A Contingency Approach Proposition 1. Pair programming is more effective than solo programming in improving quality when pairs are similarly skilled. Figure 1. Moderating Effect of Pair Skill Similarity Effort: The Moderating Effect of Task Complexity Task complexity reflects the degree to which tasks are defined, structured, and predictable and thus easily managed by means of standardized procedures (Vashdi, Bamberger, & Erez, 2013: 951). Tasks are complex when they require an unpredictable number of distinct steps and involve the processing of multiple informational cues (Vashdi et al., 2013; Wood, 1986). Developing an operating system such as OS, for example, is super complex it contains million lines of code and it takes several years to be completed. Prior research finds that complex tasks are more prone to error and require more effort and knowledge transfer (Sun, 2012; Liu & Li, 2012). Vanhanen and Korpi (2007), for example, find that PP is more effective than solo programming in reducing effort (i.e., time spent on a task in hours) for complex tasks. There is, however, a study contradicting this result. Vanhanen and Lassenius (2005) do not find that PP was most useful for reducing effort in complex tasks. However, reason dictates that PP should be more effective in reducing effort for complex tasks. Complex tasks are normally ill-structured, equivocal, and difficult to interpret and solve. When compared to low-complexity tasks, high-complexity tasks require greater demands of skills, knowledge, cognitive abilities, memory capacities, and effort (Liu & Li, 2012). A solo programmer might take several hours simply to process the information within complex tasks. A PP environment, however, allows programmers to share the heavy workload related to complex tasks through intense collaboration and discourse (i.e. two heads are better than one ) and through strategies such as task decomposition. For instance, the driver might be trying to understand one aspect of a problem while the navigator might be simultaneously working on another. This ability to share the workload and simultaneously address the issues related to complex tasks is likely to decrease total time spent on a particular task. Following this logic we assert that, all else equal, PP is more effective than solo programming in reducing programming effort when tasks are complex. Proposition 2. Pair programming is more effective than solo programming in reducing effort (time spent in hours) when tasks are complex. Figure 2. Moderating Effect of Task Complexity Twentieth Americas Conference on Information Systems, Savannah,

6 SIGADIT Learning: The Moderating Effect of Pair Compatibility PP involves social interaction between two people and so to understand PP effectiveness we must first understand pair compatibility. We define pair compatibility as the extent to which developers/students are able to feel comfortable with one another when working together on a particular task. Previous studies report that pairs, in which individuals do not feel comfortable with one another, dislike collaborative work (Layman, 2006). We assert that individuals who are uncomfortable working together are also less likely to communicate with each other during the task. This is because people tend to be more reserved around those they are not comfortable with. If pairs rarely communicate it is very unlikely that they will be learning from one another. Rare communication implies that almost no knowledge will be shared between programmers, from tips on programming languages rules to overall design skill. If almost zero knowledge is exchanged, the likelihood of learning is quite small (Upchurch & Williams, 2001). In fact, prior research finds that PP only enhances learning because pairs engage in effective communication through an open and transparent environment (Siobhan, 2007). We therefore propose the following: Proposition 3. Pair programming is less effective than solo programming in improving learning when pairs are incompatible (uncomfortable with one another when working together on a particular task.) Figure 3. Moderating Effect of Pair Compatibility Conclusion and Future Research Empirical findings to date associated with PP effectiveness are clearly mixed. In this essay we have begun theoretically unpacking these mixed outcomes through moderation arguments that highlight the importance of pair skill similarity, task complexity, and pair compatibility on differentiated PP outcomes. We have proposed that PP will be superior to individual programming for increasing code quality when pairs are similarly skilled. We further assert that all else equal, PP is more effective than solo programming in reducing programming effort when tasks are complex. Last, we propose that PP does not enhance learning when individuals in pairs are incompatible and therefore uncomfortable when working with each other. Our next step involves testing our assertions. We plan to do that in a two-phase study. First, we will conduct semi-structure interviews assessing practitioners view of PP effectiveness. Certainly, field studies do not enable researchers to isolate effects the way a controlled experiment does, but field studies have the benefits of initially investigating our contingency factors for external validity and of helping us to further theorize and identify alternative explanations. Second, we will conduct experiments using masters-level students in an evening executive program for projects that carry through two semesters. In using experienced developers the study would do more to resemble practitioner situations than the study of undergraduates. Future research should do more to unpack the contingencies where pair programming might be most effective. One clear avenue for further exploration involves taking temporality into consideration. As Cockburn (2000) observed, practitioners who originally resist pair programming often grow to appreciate the method over time. Therefore, just as managerial support is important to the adoption of any innovation, so too might management s attitude toward pair programming affect the attitudes of programmers over time about the practice. If management encourages the practice over time, programmers are more likely to stick with it. 6 Twentieth Americas Conference on Information Systems, Savannah, 2014

7 Pair Programming: A Contingency Approach Another avenue for further theorizing involves the possibilities for three-way interactions. Consider the moderating effects of pair skill level and task complexity on the effect of pair programming on cost. It might be unnecessary and costly for practitioners to pair experienced developers on simple tasks they are probably better off working on these tasks alone but both practitioners and programmers might benefit dramatically from a combined effort on complex tasks. It is also possible that extremely experienced developers those who are way above average and therefore outliers are just better off working alone and will never buy-in the idea of PP. When considering effort, practitioners might want to pair novice developers on simple tasks because the they might more efficiently generate simple programming tasks more effectively in pairs, but for complex tasks novice developers may have little effect on the total effort. If the goal is not on output quality but rather on the training and learning of novice programmers, practitioners might want to pair experienced developers with beginners in simple tasks so that the former can teach the latter some specifics about programming through PP. Other studies should also consider the effects of pair complementary skills (programming versus database) on the superiority of PP over individual programming. Twentieth Americas Conference on Information Systems, Savannah,

8 SIGADIT References Ambler, S. W Survey Says: Agile Works in Practice, Dr. Dobb s Journal: The World of Software Development (31:9), pp Arisholm, E., Gallis, H., Dybå, T., and Sjøberg, D. I. K Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise, IEEE Transactions on Software Engineering (33:2), pp Balijepally, V., Mahapatra, R., Nerur, S., and Price, K. H Are Two Heads Better Than One For Software Development? The Productivity Paradox Of Pair Programming, MIS Quarterly (33:1), pp Beck, K Extreme programming explained: embrace change / Kent Beck., Reading, MA: Addison-Wesley, Di Bella, E., Phaphoom, I., Sillitti, N., Succi, A., Vlasenko, G Pair Programming and Software Defects--A Large, Industrial Case Study., IEEE Transactions on Software Engineering (39:7), pp Canfora, G., Cimitile, A., Garcia, F., Piattini, M., and Visaggio, C. A Evaluating performances of pair designing in industry., Journal of Systems & Software (80:8), pp Cliburn, D. C Experiences with pair programming at a small college, J. Comput. Small Coll. (19:1), pp Cockburn, A., and Williams, L The Costs and Benefits of Pair Programming, Extreme programming and flexible processes in software engineering, M. Marchesi and G. Succi (eds.),, pp DeClue, T. H Pair programming and pair trading: Effects on learning and motivation in a cs2 course, Consortium for Computing Sciences in Colleges, J. G. Meinke (ed.), (Vol. 18) Presented at the Journal of Computing in Small Colleges, pp Gehringer, E. F A pair-programming experiment in a non-programming course., Companion of the 18th Annual ACM SIGPLAN Conference: Object-oriented Programming, Systems, Languages & Applications, p Han, K.-W., Lee, E., and Lee, Y The Impact of a Peer-Learning Agent Based on Pair Programming in a Programming Course, IEEE Transactions on Education (53:2), pp Hanks, B., McDowell, C., Draper, D., and Krnjajic, M Program Quality with Pair Programming in CS1, SIGCSE BULLETIN (36), pp Hannay, J. E., Dybå, T., Arisholm, E., and Sjøberg, D. I. K The effectiveness of pair programming: A meta-analysis, Information & Software Technology (51:7), pp Heiberg, S., Puus, U., Salumaa, P., and Seeba, A Pair-Programming Effect on Developers Productivity, Extreme programming and agile processes in software engineering, M. Marchesi and G. Succi (eds.), Presented at the Lecture Notes in Computer Science, pp Jensen, R. W A Pair Programming Experience., Overload (65), p. 22. Layman, L Changing Students Perceptions: An Analysis of the Supplementary Benefits of Collaborative Software Development., 19th Conference on Software Engineering Education & Training (CSEET 06), p Liu, P., and Li, Z Task complexity: A review and conceptualization framework., International Journal of Industrial Ergonomics (42:6), pp Twentieth Americas Conference on Information Systems, Savannah, 2014

9 Pair Programming: A Contingency Approach Lui, K. M., and Chan, K. C. C When Does a Pair Outperform Two Individuals?, Extreme programming and agile processes in software engineering, M. Marchesi and G. Succi (eds.), Presented at the LECTURE NOTES IN COMPUTER SCIENCE, pp Marchenko, A Pair programming. What researches say on the costs and benefits of the practice., AgileSoftwareDevelopment.com,. McDowell, C., Bullock, H., Fernald, J., and Werner, L The Effects of Pair-Programming on Performance in an Introductory Programming Course, SIGCSE BULLETIN (34), pp McDowell, C., Werner, L., Bullock, H. E., and Fernald, J The Impact of Pair Programming on Student Performance, Perception and Persistence, International Conference on Software Engineering (25), pp Nawrocki, J. R., and Wojciechowski, A Experimental Evaluation of Pair Programming, European Software Control and Metrics, pp Nosek, J. T The Case for Collaborative Programming., Communications of the ACM (41:3), pp Richardson, I., Abrahamsson, P., Messnarz, R., Nawrocki, J. R., Jasiński, M., Olek, Ł., and Lange, B Pair Programming vs. Side-by-Side Programming.,, p. 28. Rostaher, M., and Hericko, M Tracking Test First Pair Programming - An Experiment, P universe and first agile universe conference, D. Wells and L. Williams (eds.), Presented at the LECTURE NOTES IN COMPUTER SCIENCE, pp Rumpe, B., and Schröder, A Quantitative Survey on Extreme Programming Project, Presented at the 3rd International Conference on Extreme Programming and Flexible Processes in Software Engineering, Sardinia, Italy. Sanders, D Student Perceptions of the Suitability of Extreme and Pair Programming, Presented at the P Universe. Siobhan Paired Programming - Benefits for Individual and Team Development, Musings on agile, personal development and IT,. Sun, W The true cost of pair programming: Development of a comprehensive model and test., US: ProQuest Information & Learning. Tessem, B Experiences in Learning P Practices: A Qualitative Study, Extreme programming and agile processes in software engineering, M. Marchesi and G. Succi (eds.), Presented at the Lecture Notes in Computer Science, pp Upchurch, L. W., Williams, L In Support of Student Pair Programming. ACM SIGCSE Bulleting 33 (1), Van Toll, T., Lee, R., and Ahlswede, T Evaluating the Usefulness of Pair Programming in a Classroom Setting., 6th IEEE/ACIS International Conference on Computer & Information Science (ICIS 2007), p Vanhanen, J., and Korpi, H Experiences of Using Pair Programming in an Agile Project, Hawaii International Conference on System Sciences (CONF 40), pp Vanhanen, J., and Lassenius, C Effects of pair programming at the development team level: an experiment., 2005 International Symposium on Empirical Software Engineering, 2005., p. 10. Twentieth Americas Conference on Information Systems, Savannah,

10 SIGADIT Vanhanen, J., and Lassenius, C Perceived Effects of Pair Programming in an Industrial Context., 33rd EUROMICRO Conference on Software Engineering & Advanced Applications (EUROMICRO 2007), p Vanhanen, J., Lassenius, C., and Mantyla, M. V Issues and Tactics when Adopting Pair Programming: A Longitudinal Case Study., International Conference on Software Engineering Advances (ICSEA 2007), p. 70. Vashdi, D. R., Bamberger, P. A., and Erez, M Can surgical teams ever learn? The role of coordination, complexity, and transitivity in action team learning., Academy of Management Journal (56:4), pp Venkata Vinod Kumar Padmanabhuni, Hari Praveen Tadiparthi, Muralidhar Yanamadala, and Sagar Madina Effective Pair Programming Practice- An Experimental Study, Journal of Emerging Trends in Computing and Information Sciences (4), p Webster, J., and Watson, R. 2002, June. Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly, pp. xiii xxiii. Williams, L But, Isn t That Cheating?, Frontiers in Education Conference (2), pp. 12b9 26. Williams, L. A. K., Robert R Experiments with Industry s Pair-Programming Model in the Computer Science Classroom, Computer Science Education (11:1), pp Williams, L., and Kessler, R. R Pair programming illuminated / Laurie Williams, Robert Kessler., Boston: Addison-Wesley, Williams, L., Robert R Strengthening the Case for Pair Programming., IEEE Software (17:4), p Twentieth Americas Conference on Information Systems, Savannah, 2014

11 Pair Programming: A Contingency Approach Appendix Study Quality Effort Learning Abrahamsson (2005) Amber (2007) Arisholm et al. (2007) Balijepally et al. (2009) Cliburn (2003) Cockburn & Williams (2001) Declue (2003) Di Bella et al. (2013) Gehringer (2003) Han et al. (2010) Hanks et al. (2004) Hannay et al. (2010) Lui and Chan (2003) Marchenko (2008) McDowell et al. (2002) McDowell et al. (2003) Nawrocki & Wojciechowski (2001) Nosek (1998) Rostaher & Hericko (2002) Rostaher & Kericko (2002) Rumpe & Schroer (2002) Salleh et al. (2011) Sanders (2001) Sun (2012) Tessem (2003) Vanhanen and Lassenius (2005) Vanhanen and Lassenius (2007) Venkata et al. (2012) Williams & Kessler (2001) Williams (1999) Williams et al. (2000) Table 2. Concept Matrix Twentieth Americas Conference on Information Systems, Savannah,