Software Quality Improvement by using an Experience Factory

Software Quality Improvement by using an Experience Factory Frank Houdek erschienen in Franz Leher, Reiner Dumke, Alain Abran (Eds.) Software Metrics - Research and Practice in Software Measurement Deutscher Universitätsverlag, Wiesbaden: Gabler 1997 IBSN: 3-8244-6518-3 Seiten 167-182 Abstract Systematic learning in the own domain and reusing this experience is a promising way in order to achieve higher quality and productivity. An organizational approach for building competencies and supplying them to software projects is provided by the Experience Factory. An important role within the Experience Factory plays the task of packaging information into experience packages. The structure of these experience packages must support identification of potential information and the task of reuse. In this paper we identify main problems in packaging experience insufficient structure, unsuitable classification and missing technical support and introduce an approach in order to overcome these deficits. Main characteristic of this approach is a reprocessing of information beyond a problem-solution strategy. Other elements are classification schemes and a technical support by an experience base. An example and a conclusion complete this paper. 1 Introduction Software plays an essential role in our live. It controls various and sometimes safety-critical systems, like airplanes and power plants. Corresponding to this increasing importance of software there is an increasing demand on higher quality. For the software developing organizations the ability of building high quality software systems becomes more and more a strategic factor. Meanwhile it has been recognized that process improvement is an essential step to achieve high quality products. But there is a lack of knowledge, which modifications of the development process will cause which results. A promising way to manage this problem is continuous accumulation and learning of evaluated experience. To do this successfully, there must be a framework that supports and institutionalizes learning, packaging of obtained experience, and reusing of stored experience in a suitable manner. The Quality Improvement Paradigm (QIP) [BC93, GR96] and the Experience Factory [BCR94] gives a way to cope with these issues. Whereas the QIP is a framework for systematic quality improvement by learning, the Experience Factory concept describes an organizational structure that institutionalizes the tasks of packaging and reusing experience. These processed experience descriptions are called experience packages, and they are stored in an experience base [Bas93]. Within the Experience Factory, these experience packages and their structure play an important role. If they are too informal, nearly no support can be given to a user of the experience

base. On the other hand, a too formal structure would prevent the description of various kinds of software engineering experience. In this paper, we present an approach for packaging software engineering experience into experience packages to enable effective reuse. In the following section we want to explain the idea of quality improvement by (re-)using experience in more detail. After that, we present the existing approaches to build experience packages and their deficits. Then we present our approach with its main element, the so-called Quality Patterns, which overcomes existing deficits by describing experience in a problem-oriented way. Additional parts of our approach are some classifications techniques and technical support for handling Quality Patterns in a efficient and comfortable way. An example follows in a separate section. At the end, there is a conclusion and an outlook to our future work. 2 Experience Factory Continuous software quality improvement needs systematic techniques to capture and reuse all kinds of software engineering experience. This experience will vary in different domains according to their specific processes, techniques, and goals. So, software engineering experience will not be universal, but domain specific. Possible types of software engineering experience are process models, quality models, techniques, development methods, baselines, and many more. We can use this experience in projects to tailor their processes and models according to their particular needs and quality requirements. By packaging the new experience into new or refined models and experience packages, we put the idea of continual learning into action. A framework to build and reuse domain specific knowledge in a systematic and measurement based way is given by the Quality Improvement Paradigm (QIP) [BC93]. It is an adaptation of the scientific method in the context of software engineering. The concept of the Experience Factory [BCR94] provides an organizational structure to obtain information, build models and experience packages, and infuse the technology into software projects again. In the concept, the organization is devided up into two parts, the project organization, which deals with developing software, and the Experience Factory, which deals with models and experience. In figure 1 this segmentation is displayed graphically. Additionally, the flow of conceivable information is depicted, too. The three steps in the project organization box describe the planing and the execution of a project: At the beginning, the project and its context are characterized. According to the information obtained in characterizing the project, experience from similar projects is retrieved from the experience base. By assistance of this experience, a project plan is created. Normally, this project plan includes a measurement plan which defines what software metrics should be measured at what time. This will enable control of the project on the basis of information from similar projects (e.g. time and cost schedule, error detection rate,...). During the execution there is an iterative exchange of measured data and feedback by attention of the measured data and the stored experience. At the end of the project, all information and experience are packaged in a reusable form in order to build new or refined experience packages. An important part within the idea of reuse of experience, especially in the concept of the Experience Factory, plays the description of information as experience packages. A structure for them is needed which enables reusing, generalizing, and tailoring effectively. But until now, there exist only a few ideas concerned with this problem. Before we present our approach, we give a short survey about the existing approaches stated in literature.

3URMHFWÃ2UJDQLVDWLRQ ([SHULHQFHÃ)DFWRU\ &KDUDFWHUL]HÃWKHÃ SURMHFWÃHQYLURQPHQW FKDUDFWHUL]DWLRQ DVSHFWV SURMHFW HQYLURQPHQW FKDUDFWHULVWLFV 6XSSRUWÃRIÃSURMHFWV 6HWÃPHDVXUDEOHÃJRDOV &KRRVHÃSURFHVVHV JRDOVÃWRROV SURFHVVHVÃIURP VLPLODUÃSURMHFWV ([SHULHQFHÃ%DVH 3DFNDJH Ã*HQHUDOL]H Ã7DLORU Ã)RUPDOL]H H[HFXWLRQÃSODQ ([HFXWHÃWKHÃSURMHFW DQGÃFROOHFWÃGDWD GDWDÃOHVVRQVÃ OHDUQHG SURMHFWÃDQDO\VLV PRGLILFDWLRQV $QDO\VLQJÃWKHÃ FROOHFWHGÃGDWD Figure 1: Project organization and Experience Factory. 3 Related Work The task of building experience packages plays a key role within the concept of the Experience Factory. Basili [Bas93] makes some general suggestions about different kinds of experience that can be packaged into experience packages. He identifies six subtypes, e.g. process and product packages. As his main intention was to illustrate the idea of the Experience Factory, these suggestions were neither formalized nor defined in detail. Nevertheless they give a good impression of the kinds of experience which can be stored in an experience base. In the ESPRIT 1 project PERFECT (Process Enhancement for Reduction of software defects) the task of structuring experience was considered, too. The structure that was identified there consists of the following parts: context description, process models, quality models, and experience [Exp96, PEF96]. The context description contains a characterization of the environment and involved project, in which the experience was captured. In process models, there are descriptions of the involved activities, methods, and roles. The quality models represent relationships between different input factors and measurable quality aspects. The experience part consists of feedback reports, improvement recommendations, dependency diagrams, and all kinds of experience which can not be expressed in the other three parts of the experience package. This type of experience package is mainly limited to software process experience and therefore not suitable for many other kinds of experience. Beside the theoretical contributions, there are also practical results from an existing Experience Factory, the Software Engineering Laboratory of the NASA Goddard Space Flight Center [BM96]. Their experience is described in reports as plain text (e.g. [SEL95]). Often, they include measurement data in a processed way, but also diagrams, figures, and tables. 4 Quality Patterns In this section we want to introduce our approach in describing experience in the context of the Experience Factory. The basis for our work was a small survey in our organization in order to get an idea if and how experience is collected and reused at all [Fel96]. The existing way in describing experience is the writing of experience reports. Of course, this structure has a couple of disadvantages, which we categorized into the following classes: Insufficient structure of the experience reports. There is no common structure in writing experience reports, so each author implicitly de- 1 The European Union s information technologies program.

fines his or her own structure according to his or her understanding and writing style. The consequence for a future user of this experience report is that he or she must read the whole document to capture all described experience. If the experience reports are non-trivial, this can be a time consuming task. Incomplete description of experience. Normally, experience reports are written after project execution. In longer projects, a lot of valuable experience is forgotten. Another reason for incomplete description is that reports are not reviewed by others at all, in order to find incomprehensible or ambiguous explanations. Bad classification of the experience reports. Typically, there is no index about the existing experience reports available. The few existing ones are only in alphabetic or chronological order. So there is no support of a user who is looking for a particular kind of experience (e.g. efficient review techniques for specifications). By this, he or she has to look at each report and must at least read it cross over. Inadequate technical support. There is no automated support in managing experience reports. By this, mainly the task of searching for a particular experience, as described above, is very lavish. But also the task of maintaining the reports (which normally is not done at all) is very difficult. Only local access to the experience. Normally, the experience reports are located at a single place within an organizational unit (e.g. a department). So the experience is only locally available, although it is of interest to other parts of the organization, too. These disadvantages can be characterized as technical problems. Another problem, which is even worse, is the lack of motivation of the engaged people. Although there is the request to writing experience reports, there is often no time assigned to write these reports. The importance of systematic experience transfer is not really recognized by the management, but rigorous support by the management and continual motivation are absolutely necessary. The usage of formal methods and techniques can not be very useful here. Positive results, for example, were achieved at IBM in their program in code reuse by applying reward mechanisms [Kru92]. Although we are aware of these problems, we will not focus on these management aspects in this paper. Instead, we will present an approach to overcome the existing technical problems. On the left of figure 2 the different technical problems are displayed again. On the right, there are a couple of concepts that we have selected to deal with these problems. The arrows indicate which concept is used to solve which problem. At first we deal with the description of experience in face of reuse. Here we used the concepts of patterns (description of experience in pairs of problem and solution) and pyramid thinking (rearrangement of information in order to make it more comprehensible). 4.1.1 Patterns The concept of patterns is well known in literature. It was introduced by C. Alexander, an architect, in the late 1970s as a way for the documentation of experience [Ale79]. He observed the method that experts apply in problem solving. He recognized that they never search for complete new solutions but recall solutions they former found. These solutions are applied to the new problem after executing necessary modifications. So the behavior of experts can be described as thinking in pairs of problem and solution or as thinking of patterns.

Insufficient structure of the experience reports Incomplete description of experience Bad classification of the experience reports Inadequate technical support Only local access to the experience Patterns Pyramid thinking Facett classification Textual search Distributed hypermedia information system (WWW) Figure 2: Problems and identified solution concepts. Patterns are very suitable for carrying information or experience in face of reuse. The main reason for this is that patterns are problem-oriented rather than solution-oriented. In describing a pattern, the main focus is not only to present a solution, but in observing what problems a user of this experience will have in future. A very popular example of application of patterns in software engineering are design patterns [GHJ+94], which are used in object-oriented technology in order to reuse software designs. Of course, there are often limiting constrains to the applicability of problem-solution pairs due to the particular environment. So the description of the environment in which the experience was captured is also an important part in the description of a pattern. 4.1.2 Pyramid thinking Pyramid thinking [Min87] is a conceptual way in structuring information. The information is presented in information layers of increasing volume and complexity. At the top level you find a summary of the result. In the underlying layers the information of the summary is explained in more detail. So successively more information is presented. The writing of an abstract in front of an article is an example of the application of the pyramid thinking concept with two information layers. The benefit of this concept is that you can comprehend the main aspects of a document quite fast, and there is no danger to miss an important point. So the reader can decide at a very early point whether the presented information is relevant to him or her or not. The resulting structure that we got by combining these two concepts is the so-called Quality Frame, which is displayed in figure 3. Instantiations of this structure, the experience packages itself, are called Quality Pattern. The first level contains a classification of the described experience (see next subsection). In the abstract (second level) the main aspects of the described experience are mentioned. After reading this part the user has a first idea of the experience package s content. If it does not meet his or her requirements, he or she can reject it without risk, because he or she knows that nothing totally different follows in the lower layers. The next two layers contain the heart of the described experience by presenting the problem-solution pattern and the context in which the experience was captured. At this level all necessary information has been passed to the reader. 4.1.3 The levels five to eight are for explanation purpose, mainly. The example can be used to explain the potential (re-)use of the described experience. In the layer explanation there should be a description how this experience was captured. E.g. if the experience was obtained by a measurement program, the used metrics, the measured data, the analysis, and the interpretation can be described there. The next layer contains links to related experience packages, and in the last layer some administrative information is stated.

Classification Abstract Problem Solution Context Example Explanation Related Experience Administrative Information Figure 3: Quality Frame. 4.1.4 At this point we want to refer to the example in the next section, in which the Quality Frame is more clarified by presenting an exemplary Quality Pattern. After the definition of a structure and dealing by this with the problem of insufficient structure and incomplete description, now we want to present our approach in classifying experience. A number of classifications are known from literature. In the following (not exhaustive) list we want to recall them [Fel96]. Hierarchical classification. In this classification technique the elements to be classified are arranged within a tree structure. The leaf nodes are the elements and the inner nodes are decision points. If a user wants to search for a particular element, he or she starts at the root, chooses according to the annoted decision a path to a successor and iterates this proceeding until he or she stops at a leaf with the desired element. Although this is a very powerful structure, there is one main disadvantage: the user has to make a decision at each inner node, even if this decision aspect seems not relevant to him. In figure 4 there is a small example of such a hierarchical classification. commercial system System type embedded system Team size System size small large small large middle A B C D E Figure 4: Hierarchical classification. Faceted classification. Here are all elements classified by a fixed number n of attributes. Each element is assigned with n attribute values, which are drawn from a fixed attribute domain. The process of searching can be done here even with incomplete information: the users specifies m n attributes, and the retrieved elements are those which match in the specified m attributes. Example: The elements A and B are classified with (u,v,w) and (u,x,y), respectively. By classifying the desired element with (u,_,y), whereas _ indicates a do not care placeholder and therefore only 2 out of 3 attributes are specified, the result would be element B.

Relations between elements. Whereas the previous techniques provide an explicit classification, relationship between elements provides a kind of implicit classification. Elements A and B which are closely related are assigned by R(A,B) in a relation R. The other techniques would express the close relation by similar classification (nearly the same decisions in the searching tree or some equal attribute values, respectively). The problem is here, that a starting point is needed to find similar elements. Example: Consider 4 elements A, B, C, and D and a relation R with R(A,C), R(C,A), and R(C,B). Starting at element C, we could reach A and B, but not D. By this we get an implicit classification of all elements. Textual description. This technique is rather straightforward, but can be applied at text documents only. The user, who wants to search for a particular element (or document), specifies a number of words, which must appear in the wanted document. Although is technique is very simple, there are a lot of problems. If there are many (large) documents, the search can be very time consuming. Another problem is the ambiguity of natural language, which means that the same thing can be described by different words. This problems are well known, e.g. in searching for literature by using keywords. Formal approaches. In some domains, especially in the artificial intelligence, often highly sophisticated techniques, like predicate logic or algebraic specifications, are used in characterizing and classifying objects. The disadvantage of these really powerful techniques bases on the universal applicability (they are often designed for a special purpose) and their complexity (normal software developers are often not able to apply these techniques). As this survey makes clear, there is no single technique which allows a fully satisfying classification. So we decided to use a combination of different techniques, namely facet classification, textual classification, and relations between elements. At the beginning of the search process, the user selects a couple of elements by using facet classification. In an iterative manner he or she shrinks this set by using textual search and attribute refinement, until he or she has a sufficient small set of elements, which can be examined in quite short time. By using the relations he or she can get additional elements. The remaining problems are insufficient technical support and local restricted access. We address these two areas by using a hypermedia information system like the world wide web (WWW). The benefits are obvious: The handling of these technologies is common to typical users. Often, using such information systems is part of their daily work in order to get new information. This technique is wide spread and independent from concrete hard- and software. This technique supports the integration of other kinds of information and experience, so that the new system is not a replacement of existing systems but an extension. The implicit classification technique by using relations is naturally supported by hyperlinks. Based on this approach, consisting of (1) structuring experience by using Quality Frames, (2) classifying experience packages by a combination of faceted classification, textual description, and relations between elements, and (3) supporting the handling of experience packages by a hypermedia information system, we developed a prototype of an experience base [Fel96]. In figure 5 you see a screenshot of this prototype.

Figure 5: Screenshot of the experience base prototype. 5 Example of a Quality Pattern In this section we want to illustrate our approach by presenting an example. The described experience was captured within an experiment that we conducted in the Software-Laboratory at the University of Ulm, an institution with the aim to strengthen the cooperation between academia and industry. The context of this experiment was the investigation of different SA/RT approaches to examine the applicability for specifying embedded systems [BEH+96]. In figure 6 on page 9 you can see a Quality Pattern that we identified in our experiment. Those words that are printed in italics should indicate underlying hyperlinks to related documents, which describe those words more detailed. Hyperlinks within this pattern are symbolized by underlined words and common indices, whereas the source is marked only by a number, like source 1, and the target is marked by a bracket number, like target (1). E.g. comprehensible specification 1 in the description of the problem is a link to The specification are (1) in the description of the solution. As you can see, not all parts of the Quality Pattern are filled in. But this is only due lack of space. In the example part, the used specifications and requirements would be described. In related experience there should be all links to external documents replicated, like SA/RT according to Harel. Additionally, there should be also links to other quality patterns which describe similar or contradictory experience. The missing information of the last part ( administrative information ) is straightforward.

Classification Type: Process pattern Object: Specification Purpose: Creation Environment: Embedded systems Abstract: Problem: Solution: Context: Example: Explanation: Related Exp.: Admin. Inf. Viewpoint: Analyze technique: Restrictions: Project manager Experiment Tool supported The usage of the formal SA/RT specification technique according to Harel results in shorter, better readable, and unambiguous specifications the usage of the more informal technique according to Hatley and Pirbhai. Which particular SA/RT approach should be applied in creating a specification for embedded systems out of informal requirements? The approach should have the following characteristics: comprehensible spezification 1 Input: informal requirements applicable technique 2 supported by an existing tool 3 correctness should be demonstrable in a suitable way 4 SA/RT according to Harel together with the tool Statemate (3) is a suitable approach for this problem. It is more suited than SA/RT according to Hatley/Pirbhai or Ward/Mellor. Especially the concept of building hierarchical and parallel structures is very well applicable. The specifications are (1) significantly smaller 5 and thus more comprehensible 6. Validation is possible because (4) of the simulation properties 7 of this approach. SA/RT according to Harel can be applied (2) in a quite short time 8, even by inexperience developers 9. Real-time systems with the following properties: safety-critical in the sense, that wrong operation would cause critical impact not very time-critical (i.e. deviation of given time-constrains is not disastrous) mid-size systems More concrete: The specification of an American traffic light controller which can serve a wide range of user defined configurations from a simple traffic light with one lane to a complex crossing with different lanes and traffic lights. Beside a static time scheduling there is also a dynamic scheduling due to the arriving cars possible. <not filled in> In order to make qualitative and quantitative statements to the different SA/RT approaches, we conducted the following experiment. At first we tried to transform a given SA/RT specification according to Hatley/Pirbhai in a Harel-SA/RT specification. In a second step we built directly a Harel-SA/RT specification starting from the informal requirements. The given Hatley/Pirbhai-SA/RT specification consists of 1 context diagram, 7 data-flow diagrams, 5 process activation tables, and 20 minispecs and was described in 64 pages. At these two runs we made a number of observations: The Harel-SA/RT can be learned quite fast (8). The process of transformation took 160 man-hours, including the time of learning both techniques, and the involved developers were graduate students which had no experience in developing large systems (9). The directly modeled Harel-SA/RT specification is significantly smaller (5). The new developed specification consists of 3 statecharts and 13 minispecs (14 pages) and has thereby only 20% volume compared with the given specification. The directly modeled Harel-SA/RT specification is better comprehensible (6). This is mainly due smaller size, the ability of simulation, and the unambiguous semantic. The usage of formal techniques like Harel-SA/RT enables validation (7). By simulating the transformed Harel-SA/RT specification we detected a number of faults. E.g. in one case some lanes could become green whereas the crossing lanes were still yellow. A more detailed explanation can be found in [BEH+96]. <not filled in> <not filled in> Figure 6: Example of a Quality Pattern. Which specification process? Software developers first Informal requirements Spec. Hatley/ Pribhai-SA/RT Specification Harel-SA/RT Which SA/RT spezification? second Informal requirements Specification Harel-SA/RT

6 Conclusion and future work Systematic reuse provides a suitable way to achieve higher productivity in development as well as higher quality of the resulting products. Necessary for systematic reuse is a continuous process of learning, capturing, and packaging of experience as well as a framework for reuse. The concept of the Experience Factory defines such a framework by deviding up an organization into two parts, the project organization, which is concerned with the daily project work, and the Experience Factory, which collects, packages, and provides experience to the project organization. Within the Experience Factory, the experience base as a repository of experience plays a major role. In this paper, we have presented an approach to structure and store experience to enable effective reuse. The bases of this approach were the existing problems in collecting and reusing experience, which can be classified by insufficient structure, unsuitable classification and missing technical support. To overcome these deficits, we have combined a number of wellknown techniques, namely pattern languages and pyramid thinking in order to define a uniform but variable structure, the so-called Quality Frame, faceted classification, textual description, and relations between elements to classify the experience and support efficient searching, and a hypertext-based information system to enable comfortable usage and distribution of experience over all sites of an organization. In future work, there will first be a validation of our concepts and the experience base prototype by application in field. Application providers will be three different environments in industrial field within a German public funded research project, named by the abbreviation SoftQuali (see acknowledgments), from which this work was sponsored. In addition to validation some other topic will be addressed, too. One topic is a more formalized structure of the experience packages in order to enable better searching and support in generalization and adaptation of experience. Another aspect is the further development of the experience base prototype. First applications of our approach had shown, that the maintenance of a system of Quality Patterns is a very time consuming job. To give some assistance here, we want to develop a formalism to describe the various dependencies between different experience packages to generate relations between them automatically. When starting an Experience Factory, the experience base should not be empty in order to enable usage from beginning. An interesting question here is, what initial knowledge should and could be provided to the users. We will try to give some first suggestion in this direction, too. 6.1.1 Acknowledgments The research project, which serves as a basis for this paper, is funded by the German Federal Ministry of Education, Science, Research, and Technology under contract number 01 IS 518 A. It does not necessarily reflect the position or the policy of the government and no official endorsement should be inferred. The responsibility for the content of this paper remains with the author.

7 References [Ale79] [Bas93] [BC93] [BCR94] [BEH+96] [Exp96] [Fel96] [GHJ+94] [GR96] C. Alexander. The timeless way of building. Oxford University Press, New York, 1979. V.R. Basili. The Experience Factory and its Relationship to other improvement paradigms. In I. Sommerville, M. Paul (Eds.): Proceedings of the 4th European Software Engineering Conference, Garmisch-Partenkirchen, 1993, pp. 68-83. V.R. Basili and G. Caldiera. Software quality improvement Strategy and practice. Harvard Business Review, 1993. V.R. Basili, G. Caldiera, and H.D. Rombach. The Experience Factory. In J.J. Marciniak (Eds.): Encyclopedia of Software Engineering, vol. 1. John Wiley & Sons, New York, 1994. B. Biechele, D. Ernst, F. Houdek, J. Schmid, and W. Schulte. Experience in modeling embedded systems with different SA/RT approaches. University of Ulm, Technical Report, 1996 (in German). Experience Base. A booklet from the PERFECT ESPRIT project 9090 handbook edition, 1996. T. Fellger. Structuring and classification of experience knowledge with hypertextbased realization. Master s thesis. University of Mannheim, June 1996 (in German). E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design Patterns - Elements of reusable object-oriented software. Addison-Wesley, Reading, Massachusetts, 1994. C. Differding and H.D. Rombach. Continual software quality improvement in industrial practice. In C. Ebert and R. Dumke (Eds.), Software metrics in practice, Springer-Verlag, Berlin, 1996, pp. 14-44 (in German). [Kru92] C.W. Krueger. Software reuse. ACM Computing Surveys, 24, 1992, pp. 131-183. [Min87] [PEF96] [Pri91] [SEL95] B. Minto. The pyramid principle - logic in writing and thinking. Minto International Inc., London, 3rd rev. edition, 1987. The Perfect Experience Factory Model. A booklet form the PERFECT ESPRIT project 9090 handbook edition, 1996. R. Prieto-Díaz. Implementing faceted classification for software reuse. Communications of the ACM, 34 (5), 1991, pp. 88-97. Software process improvement guidebook. Software Engineering Laboratory Series, SEL-95-102, March 1996. On-line available at http://fdd.gsfc.nasa.gov/selspig.pdf.