BASICS OF SOFTWARE ENGINEERING EXPERIMENTATION

Transcription

1 BASICS OF SOFTWARE ENGINEERING EXPERIMENTATION

2 Basics of Software Engineering Experintentation by Natalia Juristo and Ana M. Moreno Universidad Politecnica de Madrid, Spain SPRINGER SCIENCE+BUSINESS MEDIA, LLC

3 A C.I.P. Catalogue record for this book is available from the Library of Congress. ISBN ISBN (ebook) DOI / Printed on acid-free paper All Rights Reserved 2001 Springer Science+ Business Media New York Originally published by Kluwer Academic Publishers in 2001 Softcover reprint of the hardcover 1st edition 2001 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permission from the copyright owner.

4 CONTENTS LIST OF FIGURES LIST OF TABLES FOREWORD ACKNOWLEDGEMENTS xi xiii xix xxi PART 1: INTRODUCTION TO EXPERIMENTATION 1. INTRODUCTION 1.1. PRE-SCIENTIFIC STATUS OF SOFTWARE ENGINEERING 1.2. WHY DON'T WE EXPERIMENT IN SE? 1.3. KINDS OF EMPIRICAL STUDIES 1.4. AMPLITUDE OF EXPERIMENTAL STUDIES 1.5. GOALS OF THIS BOOK 1.6. WHO DOES THIS BOOK TARGET? 1.7. OBJECTIVES TO BE ACHIEVED BY THE READER OF THIS BOOK 1.8. ORGANISATION OF THE BOOK 2. WHY EXPERIMENT? THE ROLE OF EXPERIMENTATION IN SCIENTIFIC AND TECHNOLOGICAL RESEARCH 2.1. INTRODUCTION 2.2. RESEARCH AND EXPERIMENTATION 2.3. THE SOCIAL ASPECT IN SOFTWARE ENGINEERING 2.4. THE EXPERIMENTATION/LEARNING CYCLE 2.5. SCIENTIFIC METHOD 2.6 WHY DO EXPERIMENTS NEED TO BE REPLICATED? 2.7 EMPIRICAL KNOWLEDGE VERSUS THEORETICAL KNOWLEDGE 3. HOW TO EXPERIMENT? 3.1. INTRODUCTION 3.2. SEARCHING FOR RELATIONSHIPS AMONG VARIABLES 3.3. STRATEGY OF STEPWISE REFINEMENT 3.4. PHASES OF EXPERIMENTATION 3.5. ROLE OF STATISTICS IN EXPERIMENTATION

5 vi PART ll: DESIGNING EXPERIMENTS 4. BASIC NOTIONS OF EXPERIMENTAL DESIGN 4.1. INTRODUCTION EXPERIMENTAL DESIGN TERMINOLOGY THE SOFTWARE PROJECf AS AN EXPERIMENT RESPONSE VARIABLES IN SE EXPERIMENTATION SUGGESTED EXERCISES EXPERIMENTAL DESIGN 5.1. INTRODUCfiON EXPERIMENTAL DESIGN ONE-FACfOR DESIGNS HOW TO AVOID VARIATIONS OF NO INTEREST TO THE EXPERIMENT: BLOCK DESIGNS EXPERIMENTS WITH MUL TPLE SOURCES OF DESIRED VARIATION: FACfORIAL DESIGNS WHAT TO DO WHEN FACTORIAL ALTERNATIVES ARE NOT COMPARABLE: NESTED DESIGNS HOW TO REDUCE THE. AMOUNT OF EXPERIMENTS: FRACfiONAL DESIGNS EXPERIMENTS WITH SEVERAL DESIRED AND UNDESIRED VARIATIONS: FACfORIAL BLOCK DESIGNS IMPORTANCE OF EXPERIMENTAL DESIGN AND STEPS SPECIFIC CONSIDERATIONS FOR EXPERIMENTAL DESIGNS IN SOFTWARE ENGINEERING SUGGESTED EXERCISES 119 PART ill: ANALYSING THE EXPERIMENTAL DATA 6. BASIC NOTIONS OF DATA ANALYSIS 6.1. INTRODUCfiON EXPERIMENTAL RESULTS AS A SAMPLE OF A POPULATION STATISTICAL HYPOTHESES AND DECISION MAKING DATA ANALYSIS FOR LARGE SAMPLES DATA ANALYSIS FOR SMALL SAMPLES READERS' GUIDE TO PART III SUGGESTED EXERCISES 151

6 Vll 7. WHICH IS THE BEITER OF TWO ALTERNATIVES? ANALYSIS OF ONE-FACTOR DESIGNS WITH TWO ALTERNATIVES 7.1. INTRODUCTION STATISTICAL SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO ALTERNATIVES USING HISTORICAL DATA SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO ALTERNATIVES WHEN NO HISTORICAL DATA ARE AVAILABLE ANALYSIS FOR PAIRED COMPARISON DESIGNS ONE-FACTOR ANALYSIS WITH TWO ALTERNATIVES IN REAL SE EXPERIMENTS SUGGESTED EXERCISES WHICH OF K ALTERNATIVES IS THE BEST? ANALYSIS FOR ONE-FACTOR DESIGNS AND K ALTERNATIVES 8.1. INTRODUCTION IDENTIFICATION OF THE MATHEMATICAL MODEL VALIDATION OF THE BASIC MODEL THAT RELATES THE EXPERIMENTAL VARIABLES CALCULATING THE FACTOR- AND ERROR-INDUCED VARIATION IN THE RESPONSE VARIABLE CALCULATING THE STATISTICAL SIGNIFICANCE OF THE FACTOR-INDUCED VARIATION RECOMMENDATIONS OR CONCLUSIONS OF THE ANALYSIS ANALYSIS OF ONE FACTOR WITH K ALTERNATIVES IN REAL SE EXPERIMENTS SUGGESTED EXERCISES EXPERIMENTS WITH UNDESIRED VARIATIONS: ANALYSIS FOR BLOCK DESIGNS 9.1. INTRODUCTION ANALYSIS FOR DESIGNS WITH A SINGLE BLOCKING VARIABLE ANALYSIS FOR DESIGNS WITH TWO BLOCKING VARIABLES 216

7 Vlll 9.4. ANALYSIS FOR TWO BLOCKING VARIABLE DESIGNS AND REPLICATION ANALYSIS FOR DESIGNS WITH MORE THAN TWO BLOCKING VARIABLES ANALYSIS WHEN THERE ARE MISSING DATA IN BLOCK DESIGNS ANALYSIS FOR INCOMPLETE BLOCK DESIGNS SUGGESTED EXERCISES BEST ALTERNATIVES FOR MORE THAN ONE VARIABLE: ANALYSIS FOR FACTORIAL DESIGNS INTRODUCTION ANALYSIS OF GENERAL FACTORIAL DESIGNS ANALYSIS FOR FACTORIAL DESIGNS WITH TWO ALTERNATIVES PER FACTOR ANALYSIS FOR FACTORIAL DESIGNS WITHOUT REPLICATION HANDLING UNBALANCED DATA ANALYSIS OF FACTORIAL DESIGNS IN REAL SE EXPERIMENTS SUGGESTED EXERCISES EXPERIMENTS WITH INCOMPARABLE FACTOR ALTERNATIVES: ANALYSIS FOR NESTED DESIGNS INTRODUCTION IDENTIFICATION OF THE MATHEMATICAL MODEL VALIDATION OF THE MODEL CALCULATION OF THE VARIATION IN THE RESPONSE VARIABLE DUE TO FACTORS AND ERROR STATISTICAL SIGNIFICANCE OF THE VARIATION IN THE RESPONSE VARIABLE SUGGESTED EXERCISES FEWER EXPERIMENTS: ANALYSIS FOR FRACTIONAL FACTORIAL DESIGNS INTRODUCTION CHOOSING THE EXPERIMENTS IN A 2k-p FRACTIONAL FACTORIAL DESIGN ANALYSIS FOR 2k-p DESIGNS SUGGESTED EXERCISES 310

8 IX 13. SEVERAL DESIRED AND UNDESIRED VARIATIONS: ANALYSIS FOR FACTORIAL BLOCK DESIGNS INTRODUCTION IDENTIFICATION OF THE MATHEMATICAL MODEL CALCULATION OF RESPONSE VARIABLE VARIABILITY STATISTICAL SIGNIFICANCE OF THE VARIATION IN THE RESPONSE VARIABLE ANALYSIS OF FACTORIAL BLOCK DESIGNS IN REAL SE EXPERIMENTS SUGGESTED EXERCISES NON-PARAMETRIC ANALYSIS METIIODS INTRODUCTION NON-PARAMETRIC METHODS APPLICABLE TO INDEPENDENT SAMPLES NON-PARAMETRIC METHODS APPLICABLE TO RELATED SAMPLES NON-PARAMETRIC ANALYSIS IN REAL SE EXPERIMENTS SUGGESTED EXERCISES HOW MANY TIMES SHOULD AN EXPERIMENT BE REPLICATED? INTRODUCTION IMPORTANCE OF THE NUMBER OF REPLICATIONS IN EXPERIMENTATION THE VALUE OF THE MEANS OF THE ALTERNATIVES TO BE USED TO REJECT H 0 IS KNOWN THE VALUE OF THE DIFFERENCE BETWEEN TWO MEANS OF THE ALTERNATIVES TO BE USED TO REJECT H 0 IS KNOWN THE PERCENTAGE VALUE TO BE EXCEEDED BY THE STANDARD DEVIATION TO BE USED TO REJECT H 0 IS KNOWN THE DIFFERENCE BETWEEN THE MEANS OF THE ALTERNATIVES TO BE USED TO REJECT H 0 IS KNOWN FOR MORE THAN ONE FACTOR SUGGESTED EXERCISES 345

9 X PART IV: CONCLUSIONS 16. SOME RECOMMENDATIONS ON EXPERIMENTING INTRODUCTION PRECAUTIONS TO BE TAKEN INTO ACCOUNT IN SE EXPERIMENTS A GUIDE TO DOCUMENTING EXPERIMENTATION 354 REFERENCES 359 ANNEXES ANNEX I: SOME SOFTWARE PROJECT VARIABLES 367 ANNEX II: SOME USEFUL LATIN SQUARES AND HOW THEY ARE USED TO BUILD GRECO-LATIN AND HYPER-GRECO-LATIN SQUARES 379 ANNEX III: STATISTICAL TABLES 385

10 LIST OF FIGURES Figure 1.1. Figure 2.1. Figure 2.2. Figure 3.1. Figure 3.2. Figure 4.1. Figure 4.2. Figure 4.3. Figure 5.1. Figure 5.2. Figure 5.3. Figure 6.1. Figure 6.2. Figure 6.3. Figure 6.4. Figure 6.5. Figure 8.1. Figure 8.2. Figure 8.3. Figure 8.4. Figura 8.5. Figure 8.6. Figure 8.7. Figure 8.8. Figure 9.1. The SE community structured similarly to other engineering communities Iterative learning process Experimentation/learning cycle Process of experimentation in SE Graph of the population of oldenburg at the end of each year as a function of the number storks observed in the same year < ) Relationship among Parameters, Factors and Response Variable in an Experimentation External parameters Internal parameters Design of the ftrst part of the study Design of the second part of the study Three-factor factorial design and two alternatives per factor Distribution of the Z statistic Student's t distribution for several values ofv Fisher's F distribution Chi-square distribution for several values of v Methods of analysis applicable according to the characteristics of the response variables Point graph for all residuals Residuals plotted as a function of estimated response variable values Residuals graph with pattern Funnel-shaped graph of residuals versus estimated values Residuals graph for each language Graph of residuals as a function of time Sample means in relation to the reference t distribution Reference distribution Distribution of residuals against estimated values for our example

11 xii Figure 9.2. Figure 9.3. Figure 9.4. Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Graph of normal probability of residuals for our example Graph of residuals by alternative and block for our example Significant language in a t distribution with the scaling factor 0,47 Domain and estimation technique effects Graph of errors and estimated values of the response variable in the umeplicated 2 4 example Residual normal probability graph Graph of residuals plotted against estimated response Graphs of effect and interaction for our example Graph without interaction between factor A and B, each with two alternatives Effects of A, B and AC Effect of the factors and interactions on normal probability paper Normal probability residuals graph Graph of residuals against estimated response Graphs of principal effects and interactions Normal probability graph of the effects of a design Graph of normal probability of the experiment residuals Graph of residuals plotted angainst predicted values for the 25.1 design described Graph of effects A, B, C and AB Graph of interaction AB Number of system releases Greco-Latin squares

12 LIST OF TABLES Table 1.1. Table 1.2. Table4.1. Table4.2. Table4.3. Table 4.4. Table 4.5. Table4.6. Table 5.1. Table 5.2. Table 5.3. Table 5.4. Table 5.5. Table 5.6. Table 5.7. Table 5.8. Table 5.9. Table Table Table 6.1. Table 6.2. Table 6.3. Table 6.4. Table 6.5. Table 7.1. Percentage of faults in the car industry Summary of fallacies and rebuttals about computer science experimentation Examples of factors and parameters in real experiments Examples of response variables in SE experiments Examples of software attributes and metrics Measurement type scales Examples of GQM application to identify response variables in an experiment Examples of response variables in real SE experiments Different experimental designs Replications of each combination of factors Temporal distribution of the observations Possible factorial design Nested design Three hypothetical results of the experiment with A and B to studyrv Suggested block design for the 2k factorial design Shows the sign for two factors and Table 5.9 shows the sign table for three factors Sign table for the 2 3 design with two blocks of size 4 2 x 2 factorial experiment with repeated measures in blocks of size 2 Another representation of the design in Table 5.10 Examples of null and alternative hypotheses Critical levels of the normal distribution for un ilateral and bilateral tests Expected frequencies according to Ho (there is no difference between tool use or otherwise) Observed frequencies Structure of the remainder of part III Data on 20 projects (using process A and B)

13 XIV Table 7.2. Table 7.3. Table 7.4. Table 7.5. Table 7.6. Table 7.7. Table 7.8. Table 7.9. Table Table Table Table Table Table Table Table Table 8.1. Table 8.2. Table 8.3. Table 8.4. Table 8.5. Table 8.6. Table 8.7. Table 8.8. Table 8.9. Table Table Table 9.1. Table 9.2. Table 9.3. Shows the 210 observations taken from the historical data collected about the standard process A Means of 10 consecutive components Difference between means of consecutive groups Results of a random experiment for comparing alternative A and B to calculations Accuracy of the estimate for I 0 similar projects Ratio of detected faults p Number of seconds subjects looked at algorithm when answering each question part Percentage of correct answers to all question parts Mean confidence level for each question part Number of seconds subjects took to answer questions Number of times the algorithm was viewed when answering each question Analysis of claim (a) Analysis of claim (b) Analysis of claim (c) Analysis of claim (d) Number of errors in 24 similar projects Effects of the different programming language alternatives Estimated values of Yij Residuals associated with each observation Analysis of variance table for one-factor experiments Results of the analysis of variance Results for good versus bad 00 Results for bad structured versus bad 00 Results for good structured versus good 00 Lines of code used with three programming languages Producting coded of 5 development tools Data taken for the example of a design with one blocking variable Effects of blocks and alternatives for our example Experiment residuals for our example

14 Basics of Software Engineering Experimentation XV Table 9.4. Table 9.5. Table 9.6. Table 9.7. Table 9.8. Table 9.9. Table Table9.11. Table Table Table Table Table Table9.17. Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Analysis of variance by one factor and one block variable Results of the analysis of variance for our example Incorrect analysis by means of one factor randomised design Coded data for 5 x 5 Latin square of our example Results of the experiment with Latin squares in our example Analysis of variance of a replicated Latin square, with replication type (I) Analysis of variance of a replicated Latin square, with replication type (2) Analysis of variance of a replicated Latin square, with replication type (3) Greco-Latin square Greco-Latin square design for programming languages Analysis of variance for a Greco-Latin design Results of the analysis of variance for the Greco-Latin square Incomplete randomised block design for the programming language experiment Results of the approximate analysis of variance with a missing datum Balanced incomplete block design for the tools experiment Analysis of variance for the balanced incomplete block design Analysis of variance for the example in Table 9.18 Data collected in a 3 x 4 experimental design Principal effects of the techniques and domain Effects of interaction a~ for our example Analysis of variance table for two factors Result of the analysis of variance for our example Experimental response variables Alternatives of the factors for our example Sign table for the 2 2 design of our example Residual calculation for our example Analysis of variance table for 2 2 design Results of the analysis of variance for our example Alternatives for three factors in our example Sign table for a 2 3 design

15 XVI Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table ll.5. Table Tabla Tabla Tabla Tabla Table Table Table Table Residual calculation Analysis of variance table for 2k model of fixed effects Values of the analysis of variance for our example Results of the specimen 2 4 experimental design Sign table for a 2 4 design Effects of the factors and interactions of our 2 4 design Residuals related to the nomeplicated 2 4 design in question Residual calculation for our example Table of analysis of variance for our example Analysis of variance for the replicated data of Table Experiment on how long it takes to make a change with proportional data Analysis of variance for the maintainability data in Table Values of n;i for an unbalanced design Values of Il;j for an unbalanced design Analysis of variance summary for (Wood, 1997) Analysis of Variance oflnspection Technique and Specification Analysis of variance testing for sequence and interaction effects Improvement in productivity with five methodologies Porcentage of reuse in a given application Effort employed Data gathered in a nested design Examples of residuals Table of analysis of variance for the two-stage nested design Analysis of variance for the data of example 12.1 Reliability of disks from different suppliers Sign table for a 2 3 Experimental Design Sign table of a design (option 1) Sign table of a design (option 2) Sign table of a design (option 3) Sign table of a design (option 4) design Result of the analysis of variance for the example design Number of errors detected in 16 program Factor alternatives to be considered

16 Basics of Software Engineering Experimentation xvii Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table 1.1. Table 1.2. Table 1.3. Table 1.4. Table 1.5. Table 1.6. Table 1.7. Table 1.8. Table 1.9. Table 1.10 Combination of alternatives related to 2 3 design with two blocks of size 4 Calculation of the effects in a 2 3 design Table of analysis of variance for k factors with two alternatives, one block with two alternatives and r replications Analysis of variance for our example Design of the experiment described in (Basili, 1996) Results of the analysis of variance for the generic domain problems Results of the analysis of variance for the NASA problem domain Data on the percentage of errors detected by the two tools Data and ranks of the CASE tools testing experiment Errors detected per time unit across nine programs Kruskal-Wallis test result for development response variables Grades attained by two groups of students Time taken to specify a requirement Lines of code with two different languages Number of replications generated according to curves of constant power for one-factor experiments Parameters of the operating characteristic curve for the graphs in Annex III: two-factor ftxed-effects model Number of replications for two-factor experiments generated using operatins curves Questions to be addressed by experimental documentation Possible values for problem parameters Possible values for user variables Possible values for information sources variables Possible values for company variables Possible values for software system variables Possible values for user documentation parameters Possible values for process variables Possible values for the variables methods and tools Possible values for personnel variables Possible values for intermediate product variables

17 XVlll Tablel.ll. Table Table III. I. Table III.2. Table Table III.4. Table III.5. Table III.6. Table Table Table Table III.lO. External parameters for the software application domain Internal parameters for the software application domain Normal Distribution Normal Probability Papel Student's t Distribution Ordinate Values of the t Distribution 90-Percentiles of the F( v 1, v 2 ) Distribution 95-Percentiles of the F( v 1, v 2 ) Distribution 99-Percentiles of the F( v 1, v 2 ) Distribution Chi-square Distribution Operating Characteristic Curves for Test on Main Effects Wilcoxon Text

18 FOREWORD Although the term "software engineering" was coined in 1968 at a NATO conference, the discipline of software engineering is still in an unfortunately prolonged adolescence. Practitioners and researchers list as major problems the same difficulties that were listed ten years ago, and ten years before that. There is very little consensus on which technologies are the most effective. Educators cannot agree on what prerequisites should be required for a computer science major, which languages should be taught, and which skills are the most valuable for good research and practice. And the major computing societies continue to bicker over what expertise is necessary for someone to be called a licensed software engineer. We need only look at other engineering disciplines to see that software development is more art than craft, and that we have a long way to go before we can rightly call ourselves "engineers." However, the picture is not completely bleak. As Natalia Juristo and Ana Moreno point out in this solid introduction to experimentation, we can learn from other disciplines whose problems are similar to ours. We can recognize that there are ways to identify possible causative factors and to organize some of our research so that we can explore and discover the effectiveness of technologies in a quantitative, reproducible way. In other words, we can and should add organized, intellectual investigation to our gut-feel decision-making about what produces the best software. That is not to say that we will fmd a one-size-fits-all approach to building good software. Indeed, we are likely to fmd that certain approaches work best in certain situations. Juristo and Moreno explain how a good experimental design will include capture of these situational factors, so that we view technological effectiveness in its organizational and human contexts. They clearly present many of the biases that are likely to affect the outcome of a study, and ways to avoid or moderate their effects, so that we see as much of the technologys effect as possible. Such valuable advice helps us to evaluate a technique or tool in our own backyards, reproducing a study to see how the technique or tool fares with our very own practices and projects. They also point out that our studies of effectiveness must be objective, where the creator of a new technique is not the only one evaluating it. Their underlying subtext is that good software engineering experimentation takes into account the ethics as well as the activities of an investigation. If you are a researcher, you should master the approaches to empirical software engineering described by Juristo and Moreno. Just as any chemist for physicist

19 XX knows how to collect and analyze data to confirm or refute underlying theories, you too should use these quantitative techniques to guide your investigations. Moreover, when other researchers follow the recommended approaches, you will have an easier time replicating existing studies and devising follow-on studies that expand what we know about software development and maintenance. If you are a practitioner, the advice in this book will enable you to read and assess the studies you fmd in your journals and at your conferences. What are the most promising technologies for the kind of software you develop? For the constraints under which you develop it? What are the trade-offs involved in adopting new technology? And how can you create or replicate studies to verify that what you read about really happens on your projects? Finally, if you are an educator, this book will help you to guide your students in understanding that software engineering is far more than simply having a good technology idea and trying it out on a project. They will see that software engineering can truly be engineering, built on a foundation of knowledge based on careful, repeatable studies. They will learn that software engineers do more than build products; they design and build products with confidence and quality. Shari Lawrence Pfleeger August2000

20 ACKNOWLEDGEMENTS So many people have provided help and support in one way or other to write this book. If we list them we could take the risk to forget somebody. We would like to thank all of them. In particular, we are specially indebted to those many who have argued directly with us; sharing their visions and considering their helpful comments have really improved the ideas that we present in this book.