Lecture Empirical Model Building and Methods (Empirische Modellbildung und Methoden) Dr. Andreas Jedlitschka SS 2016 Chapter 1 Introduction
Chapter objectives At the end of this chapter, you should have an overview of The motivation for the scientific approach to model building in software engineering Why? For what? The purpose and relevance of software engineering Why? What? For what? Rombach, Jedlitschka 2016 Slide 3
Outline 1.3 Why empirical methods? Rombach, Jedlitschka 2016 Slide 4
Why model building? Rombach, Jedlitschka 2016 Slide 5
Why model building???!?? f (?,..,?) Rombach, Jedlitschka 2016 Slide 6
A1. COCOMO Example: COCOMO - Constructive Cost Model (B. Boehm, 1981) Determination of parameters (by regression on historical project data) E = A * L B * I(X) Level Basic Intermediate Project Type A B A B organic 2,40 1,05 3,20 1,05 semi-detached 3,00 1,12 3,00 1,12 embedded 3,60 1,20 2,80 1,20 Time and required personnel are estimated using the basic model Determination of software development effort multipliers (15 attributes) I(X) = RELY * DATA * CPLX * TIME * STOR * VIRT * TURN * ACAP *AEXP * PCAP * VEXP * LEXP * MODP * TOOL * SCED Ratings (Example) Cost drivers Very low Low Nominal High Very high Extra high Product attributes RELY Required 0.75 0.88 1.00 1.15 1.40 sw reliability CPLX Product 0.70 0.85 1.00 1.15 1.30 1.65 complexity.. Rombach, Jedlitschka 2016 Slide 7
Organizational Context: EF Product Goals and Characteristics Problem/ Rqmts Project Planning Project Plan Project Management... U Req. Quality Assurance Progr. Project Organisation n Project Organisation 1 Exec. Unit SW System/ Product External Experience Processmodels Reuse (Models) Experiments Product models Resource models Experience Factory??? T/M/T Products Project plans Storage Reuse Products data... Storage (products, measures) Experience database Project database Rombach, Jedlitschka 2016 Slide 8
Model building Can quality be guaranteed by appropriate models? Models capture a kind of experience; if things work for a while, it does not mean they work for ever / or always WHY? Rombach, Jedlitschka 2016 Slide 9
Model building Models are usually only an imperfect representation of the reality Several sources of errors Theory Model building Formalization Incompleteness of factors used to describe the model Unknown factors Incomplete and incorrect data Reliability of measurement We need a systematic, retraceable way to get there Science Rombach, Jedlitschka 2016 Slide 10
Model building What is science? Science is a process to understand the world A process in search of understanding the world. A process aiming to generate knowledge about how the world works. We cannot perceive laws directly through our senses Two activities are necessary Systematic objective observation Inference of links between cause & effect Danger: Could lead to wrong conclusions We incrementally build models capturing our experience We systematically evaluate the models We need a systematic, retraceable way to observe and analyze Empirical and Statistical methods Rombach, Jedlitschka 2016 Slide 11
Model building Differentiation Reduction Aggregation Decomposition Abstraction Model Rombach, Jedlitschka 2016 Slide 12
Why empirical methods? Software part of a system that can be encoded to execute on a computer as a set of instructions; it includes all the associated documentation necessary to understand, transform and use that solution the collection of computer programs, procedures, rules, and associated documentation and data (IEEE) Software engineering the disciplined development and evolution of software systems based upon a set of principles, technologies, and processes the systematic approach to the development, operation, maintenance, and retirement of software (IEEE) the application of science and mathematics by which the capabilities of computer equipment are made useful to man via computer programs, procedures, and associated documentation (Boehm) the application and tailoring of techniques, methods, and life cycle models to the software problem, project, and organization Rombach, Jedlitschka 2016 Slide 13
Why empirical methods? The nature of software engineering Software engineering is development not production The technologies of the discipline are human based Software is inherently complex to build and understand part of the system solution we least understand often something new requirement for change/evolution of function or structure There are not any software systems that are exactly the same there are a large number of variables that cause differences their effects need to be understood Currently, insufficient set of models that allow us to reason about the discipline lack of recognition of the limits of technologies for certain contexts there is insufficient analysis and experimentation Rombach, Jedlitschka 2016 Slide 14
Why empirical methods? The nature of software engineering Like other disciplines, software engineering requires the cycle of model building, experimentation, and learning, re-modeling Software engineering is a laboratory science The researcher s role is to understand the nature of the processes, products and their relationship in the context They need laboratories to observe and manipulate the variables The practitioner s role is to build improved systems, using available knowledge They need to better understand how to build better systems More than the other disciplines these roles are symbiotic Rombach, Jedlitschka 2016 Slide 15
Why empirical methods? Real World Model World f 1..n (?,..,?) e.g., model based testing e.g. tester experience level = high, domain = automotive, programming language = C++ Variable f (Process, Context) e.g., reliability of the produced component determined by Empirical study Simulation Rombach, Jedlitschka 2016 16
Why empirical methods? Needs Systematic understanding, improvement and evaluation of project and process performance. Selection among several approaches (e.g., techniques or tools) promising to solve a given problem E.g., ~200 approaches for requirements engineering exist today +/ 5% emp Q f(p, {Context}) P = Process, Method, Technique, Tool, Q = Quality attribute / Schedule / Cost Example: +/ 5% emp Q[i] f(re[i], {Context}) search i for RE with Max(Q) Symptoms Something is wrong within software development! What? How? Why? Rombach, Jedlitschka 2016 Slide 17
Quality Models Example Process Quality model: Defect removal cost Defect removal costs grow (exponentially) with retention time (i.e., the time the defect stays in system) E.g., requirements defects cost 100 x more than coding defects Rombach, Jedlitschka 2016 Slide 18
Why empirical methods? How can empirical methods support model building in V-Model of Empirical SE research SoP: Problem Stmt. with Improvement Hyp. Emp. Testing of Problem hypotheses? SoA: Solution Stmt. with Improvement Hyp. Emp. Testing of solution hypotheses? Research Technical Solution Discipline requirements Rombach, Jedlitschka 2016 Slide 19
Empirical Software It requires the scientific use of quantitative and qualitative data to understand and improve the software product, software development process and software management Empirical means based on observation Mary Shaw 1 differentiated it from other techniques that can be used to validate research results, such as: Persuasion I have thought hard about this, and I believe that Implementation (existence proof) Here is a prototype of a system that exists in code or other concrete form Analysis Given the facts, these consequences 1 Mary Shaw, The Coming-of-Age of Software Architecture Research (keynote address). Proceedings of the 23rd International Conference on Software, Toronto, Canada, IEEE Computer Society, 2001, pp. 656-664a Rombach, Jedlitschka 2016 Slide 20
Empirical Software Conducting empirical studies requires a process prescribing steps to be performed for any empirical study, a research strategy determining how to instantiate steps for a particular study i.e., experiments, case studies, surveys Quality Improvement Paradigm (QIP) provides the basis Study results are made reusable Packa-ge Characterize Study is planned Analyze Set goals Experience factory Execu-te Choose models Study is performed Rombach, Jedlitschka 2016 Slide 21
Empirical Software Evolving knowledge Understanding a discipline involves building models, e.g., application domain, problem solving processes Checking whether our understanding is correct involves testing our models experimentation Analyzing the results of empirical studies involves learning, the encapsulation of knowledge and the ability to change and refine our models over time The understanding of a discipline evolves over time Knowledge encapsulation allows us to deal with higher levels of abstraction This is the paradigm that has been used in many other fields, e.g., physics, medicine, manufacturing,. Rombach, Jedlitschka 2016 Slide 22
Empirical Software Empiricism in several disciplines What do these other fields have in common? They evolved as disciplines when they began applying the cycle of model building, experimenting, and learning Began with observation and the recording of what was observed Evolved to manipulating the variables and studying the effects of change in the variables What are the differences of these fields? Objects they study Properties of those object Properties of the system that contain them Relationship of the object to the system Culture of the discipline This influence how the models are built how the empirical studies gets done Empiricism in other disciplines Rombach, Jedlitschka 2016 Slide 23
Empirical Software Problems of interest Practitioners want the ability to control and manipulate project solutions based upon the environment and goals set for the project knowledge based upon empirical and experimental evidence of what works and does not work and under what conditions Researchers want to understand the basic elements of the discipline, e.g., products, processes, and their characteristics (build realistic models) the variables associated with the models of these elements the relationships among these models Researchers need laboratories for observation and experimentation This requires a research plan covering several years coordinating empirical studies evolving with new knowledge State of empiricism in SE Rombach, Jedlitschka 2016 Slide 24
Research Plan Literature Survey / Postmortem analysis For instance, iterative development and evaluation of a new technology Proposal of introducing new technology Experiment Evaluation in artificial setting/ laboratory Replication Case Study Evaluation in real setting Multiple CS / Survey Large scale evaluation [Extended from Wohlin et al., 2000] Rombach, Jedlitschka 2016 Slide 25
Why empirical methods? Empirical methods Systematic approach for collecting observations and experiences Survey Interviews Observations Case studies Experiments Document analysis Focus groups Rombach, Jedlitschka 2016 Slide 26
References Basili, V.R.; Caldiera, G.; Rombach, H.D.: Experience Factory. In: Marciniak J.J. (ed.), Encyclo. of SE, Vol.1, John Wiley & Sons, 2001, pp. 511-519. Basili, V.R.; Caldiera, G.; Rombach, H.D.: Goal Question Metric Paradigm; in: Marciniak J.J. (ed.), Encyclo. of SE, Vol.1, John Wiley & Sons, 2001, pp. 528 532. Basili, V.R.; Rombach, D.; Schneider, K.; Kitchenham, B.; Pfahl, D.; Selby, R.W. (Eds.): Empirical Software Issues: Critical Assessment and Future Directions, Intern. WS Dagstuhl Castle, Germany, Jan. 2007, Springer LNCS 4336. Basili,V.; Selby, R.; Hutchens, D.: Experimentation in Software ; IEEE Trans. on SE, vol. 12(7): 733-743, July 1986 Kitchenham, B.A.; Dybå, T.; Jørgensen, M.; Evidence-based Software ; In Proc. of 26th Intern. Conf. on Software (ICSE 04); May 2004; Edinburgh, Scotland, United Kingdom, 2004, pp. 273-281. Rombach, H.D.; Basili, V.R.; Selby, R.W. (Eds.): Experimental Software Issues: Critical Assessment and Future Directions, International Workshop Dagstuhl Castle, Germany, September 1992, Springer Verlag, LNCS 706. Rombach, D.: Empirical Software Models: Can They Become the Equivalent of Physical Laws in Traditional? Int. J. Software and Informatics 5(3): 525-534 (2011) Rombach, Jedlitschka 2016 Slide 27