Validating an Evaluation Framework for Requirements Engineering Tools

Validating an Evaluation Framework for Engineering Tools Raimundas Matulevicius Dept. of Computer and Information Science, Norwegian Univ. of Science and Technology Sem Sælands vei 7-9, NO-7491 Trondheim, Norway raimunda@idi.ntnu.no Abstract. Automated support for the requirements engineering (RE) process is a recognized research area. However the mainstream practice still relies on word processors and drawing tools rather than the requirements engineering tools (RETs). The aim of this paper is to validate an evaluation framework for RETs. The validation process concerns a RET acquisition process for concrete organizational needs. An observation of maintaining requirements specification shows the important organizational and environmental characteristics for a proper automated support of RE process. The contribution of this work is twofold: first, the validation of the evaluation framework for RETs according to environmental needs in a specific environment, and second the identification of environmental needs, which emerge from requirements specification maintenance process. 1. INTRODUCTION engineering (RE) is the part of software engineering that concerns real world goals, functions, and constraints of software systems. RE is also a relationship between these factors and a precise specification of software behavior [33]. engineering tools (RETs) affect - 1) the process quality because the tools support a large part of the software engineering, in particular RE, and 2) the product quality because output of the RE is a requirements specification, which itself should be of high quality for subsequent software engineering stages. Although the need of automated support for the software development processes is recognized in the literature [9, 11, 14, 25], the mainstream of RE practice still relies on word processors, drawing and modeling tools rather than targeted tools provided by various researchers and practitioners. Current COTS (commercial off-the-shelf) for RE provide capabilities for documenting requirements and facilitating requirements management. The tools are well suited for managing large amounts of requirements written in natural language, but not for engineering the requirements [11, 12]. Many RETs are described as CASE (Computer Aided System Engineering) tools. In the early 1980s RE seemed to be a relatively simple task and existing CASE tools were expected to provide task related support for software developers. But soon weaknesses of CASE tools were discovered [13] and product and process quality improvement by using CASE tools was questionable. One plausible reason for this is the lack of maturity [11] to adopt tools. Another is the apparent cost of adopting, using and maintaining a tool. The third is the inadequate technological sophistication of the CASE tools [13]. Fitting the RETs to meet customer requirements remains problematic because companies employ different methods. Further more, RETs vary in their level of support for RE activities [10]. Thus, evaluation for selection purposes has to be performed before buying anything. A company cannot base evaluation on its own long-term tool use. Instead, it can only rely on tool surveys, commercial reports, which are unreliable and becomes quickly out of date. Evaluation of RETs differs depending on the environment, needs and purposes for tool usage. There are two core questions during the evaluation of available software tools [1]. First, how are the tools of a given domain described in order to make their comparison feasible? Second, how may the features of the tool be reconciled with respect to requirements on tools? The evaluation and comparison would be more complete and structured, if an evaluation framework, which targets such questions, is applied. For vendors of RET the evaluation framework might help to pinpoint aspects, where their tools are weak and should be improved. For a 1

RET buyer, the evaluation framework might provide a systematic evaluation and help during the decision making process. Such an evaluation framework is presented in [22]. The framework is based on analytical arguments, but not on empirical investigation. The purpose of the current work is twofold. First, we show how the framework is applied for organizational needs in a particular environment and provide results of the RETs evaluation. Second, the analysis of the requirements specification and maintenance shows requirements for an automated requirements specification support. The paper is structured as follows. Section 2 analyses the related work. Section 3 presents the evaluation framework [22] and its coverage by using a semiotic quality framework. Next, section 4 considers the process of framework validation in three steps. First, section 4.1 describes the environment and the problem, for which the evaluation framework is considered. Next section 4.2 presents the preparation phase, which consists of acquisition of finding the most important organizational requirements for the RETs acquisition and the survey of the RETs candidates. Finally, section 4.3 analyses the maintenance process of the requirements specification. Concluding remarks show important issues for the evaluation framework validity. Section 5 provides a discussion about the conclusions, and future work. 2. RELATED WORK management in the literature [7, 10, 11, 14] is described as the part of RE process and manages changes of system requirements. We use the terms requirements engineering tools (RET) instead of requirements management tools as vendors usually call these tools. The functionality of those tools covers requirements engineering activities, such as elicitation, analysis, negotiation and validation, not only management of project changes. Study of [1] emphasizes to use taxonomies for the problem domain description. INCOSE [11] suggests a taxonomy (figure 1) for RETs, based on functional characteristics. The taxonomy separates among requirements generation tools, requirement traceability tools, requirement classification tools, requirements capture tools, requirement identification tools and requirements elicitation tools. INCOSE [11] also provides a framework for evaluation of RETs and a RET survey. However, the survey is based on vendor information and in such a way is not verified for the purpose. RETs surveys at certain time intervals are provided in [18, 31]. But static tool surveys have little long-term value, as new tools are being created and features of existing ones are being continuously improved. engineering tools management tools generation tools classification tools capture & identification tools traceability tools Textual requirements capture tools Tools for elicitation of requirements Figure 1 SE Tools Taxonomy - Engineering Tools 2

A framework for comparison of modeling tools is described in [24]. Interestingly, [24] uses experts to validate the approach and to find out important requirements for an evaluation during modeling tool acquisition. But usability and adaptability of both frameworks [11, 24] is questionable, since there is no coordination or descriptions for applying the evaluation frameworks for particular needs. A methodology for describing the quality factors of software packages using ISO/IEC quality standard as a framework is introduced in [8]. Authors show that selection of packages can be ameliorated by transforming user quality requirements into requirements expressed in terms of quality model attributes. Lang and Duggan [19] suggest a list of functional requirements for requirements management, communication and cooperative work systems. But the requirements are not systematically organized and they are not complete for the RETs evaluation. Pohl s framework [26] provides the structure of the RE process. A semiotic quality framework [16, 20] identifies the main quality types of conceptual modeling. However, both Pohl s [26] and the semiotic quality frameworks [16, 20] are abstract and do not detail the requirements for the evaluation of RETs. This paper considers the framework for evaluation of RETs [22], which is constructed according to an analytical literature study. The purpose of this work is the empirical investigation and validation of the framework. The paper shows characteristics for an automated support of the RE processes. The section 3 describes the evaluation framework in detail. 3. EVALUATION FRAMEWORK FOR REQUIREMENTS ENGINEERING TOOLS A framework for RETs is described in [22] and presented in figure 2. It is based on Pohl s three-al orthogonal framework [26] and Land/Duggan requirements [19]. The framework features are requirement categories, which should be analyzed during the RET evaluation and acquisition. Each category of requirements is followed with a list of activities (table 1.), which should be tested during RET evaluation process. Framework for Evaluation of Functional for RET FEF1. Representation FEF2. Agreement FEF3. Specification FEF1.1. Specify uniquely identifiable description using informal language. FEF1.2. Specify requirements using semi- formal language(s). FEF1.3. Specify requirements using formal language(s). FEF1.4. Define traceable associations between requirements and the different elements of requirements specification. FEF1.5. Connect seamlessly with other tools and systems, by supporting FEF2.1. Maintain an audit trail of changes, archive baseline versions; and engage a mechanism to authenticate and approve change requests. FEF2.2. Classify requirements into logical user- defined groupings. FEF2.3. Support secure, concurrent cooperative work between members of a multidisciplinary team, which may be geographically distributed. FEF2.4. Maintain a comprehensive data Figure 2 Framework for Evaluation of Functional for RET FEF3.1. Collect and store common system s and product family s domain requirements. FEF3.2. Generate predefined and ad hoc reports, documents that comply with standard industrial templates, with support for presentation-quality output and in-built document quality controls. FEF3.3. Generate the complete specification, expressed using formal language (informal and semiformal languages might also be 3

RETs evaluation framework is covered by semiotic quality framework of conceptual modeling [16, 20]. There are two basic quality means on the physical level: externalization and internalizeability. A RET should support basic database functionality using a repository solution for the internal representation of the requirements model. It should deal with functionality such as version control and configuration management and advance concurrency control mechanism. Representation Agreement Specification Feature s FEF1.1 FEF1.2 FEF 1.3 FEF 1.4 FEF 1.5 FEF 2.1 FEF 2.2 FEF 2.3 FEF 2.4 FEF 3.1 FEF 3.2 FEF 3.3 Activities How does the RET FEF 1.1.1 provide natural language description. FEF 1.1.2 allow specifying unique identification (ID) for each separate requirement. FEF 1.1.3 allow importing of requirements and their description from textual document. FEF 1.2.1 provide tools for semiformal language description (f.e. ER-diagrams, UML diagrams, DFD, OMT). FEF 1.2.2 provide forward/ backward traceability between informal, semiformal, formal descriptions. FEF 1.3.1 provide tools for formal language description (f.e. Z-schemas, algebraic specifications, action semantics). FEF 1.3.2 provide forward/ backward traceability between informal, semiformal, formal descriptions. FEF 1.4.1 provide functions for testing traceability between informal, semiformal and formal requirement description. FEF 1.4.2 create parent-child traceable relations between requirements. FEF 1.4.3 maintain peer-to-peer traceable relations between requirements. FEF 1.4.4maintain traceable relation between different related information. FEF 1.4.5 maintain forward/ backward traceability between source of requirements, requirements and design. FEF 1.5.1 allow importing/exporting requirements description from/to textual documents. FEF 1.5.2 allow importing/exporting requirements description from/to graphical documents. FEF 2.1.1 maintain user authentication to the system (f.e. user name, password). FEF 2.1.2 allow grouping users into different groups. FEF 2.1.3 allow creating different views (according to documents, requirements, attributes) for different groups of stakeholders. FEF 2.1.4 register agreement/ rationale/ discussion/ negotiation/ changes/ history of requirements and by how it was achieved. FEF 2.1.5 call the earlier requirement description/ versions and register them into history context. FEF 2.2.1 allow specifying attributes/ properties of the requirement. FEF 2.2.2 provide sorting according to different attributes/ properties. FEF 2.2.3 provide filtering according to different attributes/ properties. FEF 2.3.1 provide www-based interface for geographically distributed users. FEF 2.3.2 allow making copy for modification of already approved version of requirements description in different abstract levels (document, requirement). FEF 2.3.3 provide change approval cycle for multiple change negotiation and approval before posting into common repository. FEF 2.4.1 provide the single repository or data dictionary. FEF 2.4.2 provide separate data dictionaries for non-technical users and technical users. FEF 2.4.3 provide the help system to the users. FEF 3.1.1 enable selection and extraction of common domain requirements. FEF 3.1.2 incorporate common requirements to concrete project. FEF 3.1.3 adapt/ spread changes in domain requirements to concrete projects within domain. FEF 3.1.4 provide comparison of domain requirements feasibility. FEF 3.2.1 provide wizards for report generation. FEF 3.2.2 provide possibility to print report according views and sorting. FEF 3.2.3 provide possibility to print results of rationale, brainstorm and etc. FEF 3.2.4 provide techniques for error checking. FEF 3.3.1 correspond to standards of software documentation. FEF 3.3.2 support formal languages for complete, commonly agreed requirements specification. Table 1 Activities of evaluation framework for RETs Empirical quality deals with error frequencies when a model is read or written by different users, also applies to the coding and ergonomics of computer-human interaction for modeling tools. 4

Syntactic quality has the goal of syntactic correctness. descriptions should be completed according to the syntax and vocabulary of the language. A RET should provide the means for error prevention and error detection, which may help to prevent syntactic invalidity and incompleteness errors. Semantic quality is the correspondence between the model and the domain. A RET should provide the means to reach semantic goals - feasible validity and completeness. Some of them could be consistency checking, the use of driving questions or baselines to improve completeness of the requirements specification. Perceived semantic quality is the similar correspondence between the participant s interpretation of a model and participant s current explicit knowledge. To achieve the goals of perceived validity and completeness a RET should provide the means for participant training, discussions, statement insertion and deletion. Pragmatic quality is the correspondence between the model and audience s interpretation of it. In order to achieve the goal of feasible comprehension, a RET should provide the means for requirements inspection, visualization, filtering, explanation, execution, simulation, and prototyping. Social quality deals with participant knowledge including social and technical audience interpretation. Main activities for achieving feasible agreement goal are model integration and conflict resolution: like pre-integration, viewpoint comparison and conforming, merging and restructuring. The semiotic quality framework for the RETs evaluation framework is summarized in table 2. The requirements specification should be of high quality for further software development stages. Davis et al [5] summarizes the work on quality attributes for a software requirements specification, giving the most comprehensive list of the properties. Thus [15] shows the relationships between the semiotic quality framework and the specific quality attributes, described in [5]. Quality Framework RET evaluation framework Representation Agreement Specification Physical quality Empirical quality Ext. Int. Min. err. Freq. Syntactic quality Semantic quality Correct. Valid. Comp. Perceived semantic quality Perc. valid. Perc. compl. Pragmati c quality Compr. Social qualit y Agr. FEF 1.1 FEF 1.2 FEF 1.3 FEF 1.4 FEF 1.5 FEF 2.1 FEF 2.2 FEF 2.3 FEF 2.4 FEF 3.1 FEF 3.2 FEF 3.3 Table 2 Coverage of RET evaluation framework by semiotic quality framework Ext. externalization, Int. internalizability, Min. err. freq. minimal error frequency, Correct. - syntactic correctness, Valid. validity, Comp. completeness, Perc. valid. - perceived validity, Perc. compl. - perceived completeness, Compr. comprehension, Agr agreement. 4. EMPIRICAL VALIDATION OF EVALUATION FRAMEWORK FOR RETS 5

Empirical validation of the evaluation framework falls into two parts. First, the evaluation framework is applied to organizational needs. Second, an observation of preparation requirements specification shows requirements for the automated support of the RE processes. 4.1. Environment and Problem Definition The purpose of the study was to prepare the requirements specification for an information system which is used for teaching purposes. The system registers two types of users: students and student assistants (studassist). Students are users, who submit their solutions to the system. StudAssist evaluate the solutions and form reviewer groups. The reviewer groups consist of 3 to 5 students, whose solutions are accepted by studassist. The next step is the review process. The reviews are done according to semiotic quality framework [20]. If the review results are essentially different, the studassist rejects the review, and the reviewer groups should evaluate the work again. Otherwise, the review is accepted and the results are sent to the author. The case included 216 graduate students and 6 undergraduate students as studassist, who were students at the Norwegian University of Science and Technology. Preparation of requirements specification for the system includes participants, who at different system development stages carry different roles. The process included the following stakeholder groups: organizing actors, like teachers and supervisors; leading actors, like teaching assistants; developing actors, like undergraduate students, who are responsible for the system developing and testing. In order to discover the environmental characteristics quantitative analysis was carried. Fifteen researchers were asked to fill in a questionnaire to discover the need of environment. The major research interest of participants includes information systems, workflow analysis, semantic web, implementations of decision support systems and intelligent agents. Next the investigation of the RETs candidates was performed by two evaluators. 4.2. Preparation phase The objective of preparation phase is to find the appropriate RET for maintenance of requirements specification. The evaluation framework [22] was applied in order to discover the most important aspects of the environment. Next we evaluated the set of RETs candidates. 4.2.1. Discovering environment needs The quantitative analysis includes questionnaire, which evaluates the importance of the evaluation framework [22] features. The participants were asked to evaluate the importance in the scale of 0...10 (0 - feature is not important at all; 10 - very important and useful feature). The feature importance is evaluated as a mean measure: n 1 M j = e ij n i= 1 here i is a participant index, j is a feature index, e ij is an evaluation of i participant for j feature, n a number of participants. The agreement about a singular feature is evaluated as the variance measure: n 1 2 V j = ( eij M j ) n 1 i= 1 If the variance is relatively low, it means strong agreement about the singular feature. And if the variance is relatively large, it indicates, that participants disagree about a feature. The agreement ranges from 1,4 to 8,8. In order to determine the agreed features the threshold t=5 is set. Threshold removes the features about which respondents do not agree (table 3). Participant were also asked to suggest the features to the framework, which they feel are needed to evaluate. 6

4.2.2. Evaluation of RETs The list of RETs candidates for evaluation is selected from commercial requirements engineering tools and this includes: Caliber-RM Web v.4.0. [2], Core 3.1 Trial [3], Cradle-4 [4], DOORS 5.2. [6], RequisitePro [27], RDT Version 3.0 [28], Vital Link [30], XTie-RT 3.1. [32]. Trial, demonstration and evaluation versions are evaluated according to manuals and documentation. RETs are also tried out on small examples. The survey of the evaluation is presented in [22], where RETs features are evaluated as: High=3 (very good), Medium=2 (average), Low=1 (poor). The overall evaluation E j of the RET j could be calculated as the sum of multiplications between two values: feature coverage C ij, (where i is a number of the feature) and feature importance M i, discovered during the quantitative analysis: E j = CijMi i The overall evaluation of the RETs is shown in table 3. Fe ature Agreeme Importance RETs s nt RET1 RET2 RET3 RET4 RET5 RET6 RET7 RET8 FEF1.6-10 2 1 2 1 2 2 2 2 FEF1.7-10 1 1 1 1 1 1 1 1 FEF1.2 1,4 8,3 3 2 2 3 1 1 1 2 FEF2.3 1,9 8,3 1 2 2 2 1 1 1 1 FEF3.2 1,9 8,3 2 3 2 3 3 2 2 2 FEF2.1 2,6 8,2 2 2 3 3 2 2 2 2 FEF1.1 2,8 8,1 3 2 2 2 3 2 3 2 FEF1.5 3,0 7,9 2 2 2 3 1 2 2 3 FEF3.3 3,4 7,9 1 1 1 2 1 2 2 3 FEF2.4 3,6 7,6 1 3 2 2 2 1 2 1 FEF1.4 3,6 7,5 3 2 1 2 2 2 3 3 FEF2.2 4,3 7,4 2 3 2 3 3 3 3 2 FEF3.1 4,6 7,3 1 2 2 2 1 2 2 3 FEF1.3 8,8 6,5 - - - - - - - - Overall evaluation: 196,4 209 196,4 233,7 187,7 186,8 210 218,3 Table 3 Overall evaluation of RETs according to organizational needs in particular environment FEF1.6 Define and maintain requirements constraints. FEF1.7 Allow requirements definition in the abstract level. FEF1.6 and FEF1.7 are features, suggested by participant during the quantitative analysis. FEF1.3. falls beyond the threshold. 4.2.3. Evaluation results The results of the quantitative analysis and the RETs evaluation showed how to adapt the framework [22] to the environment. First, the quantitative analysis allocates weights to evaluate the functionality of RETs. Second, it validates the RETs evaluation framework features. Third, the quantitative analysis discovers features, which are important in an environment, but not mentioned in the framework. Finally, the quantitative analysis shows the usability issues of the evaluation framework. Questioning was performed in a relatively short space of time (13 minutes in average). The framework is easily applicable to the environment. The evaluation results suggest some RETs as possible candidates (RET4, RET8, RET7, RET2). However, the evaluation showed the limitations of the RETs to solve RE problems [14]: Lack of stakeholder involvement. None of the available RETs are ideally suited for a use by a multidisciplinary, distributed team where the stakeholders have diverse skills and needs. Stakeholder communication problems - Most of RETs are standalone applications and do not provide any (or provide weak) possibilities for collaboration work and communication between stakeholders. 7

Business needs are not considered - The RE process is seen as a technical rather than a business process and it is dominated by the technical concerns. Lack of defined responsibilities - RETs should provide possibilities to define scenarios of activities for each individual participant of the RE process and let people to understand the individual responsibilities. Lack of requirements management - If the RE process does not include effective techniques or methods, it may be introduced in ad hoc way. But RETs, like CASE tools, they operate according to the method, which is defined as a set of rules, which guides the usage of a RET. RETs usually deal only with informal (in some cases semi-formal) representations of RE processes and software requirements. Automated RE support is not sufficient both to separate activities of RE and to the management of the RE processes. Because of these limitations, none of the evaluated RETs were chosen. 4.3. Execution phase The objective of execution phase is to prepare and maintain the requirements specification for an information system, used during the teaching course to evaluate students practical exercises. This requirements specification was maintained by using standard office tools, modeling tools and graphical packages. The maintenance of requirements specification shows the list of shortcomings, needed for an automated support of the RE processes, and which are covered by the evaluation framework [22]. The observed shortcomings are: Lack of automatic generation of standard requirements specification (validates the feature FEF3.3). Such requirements specification should correspond to standards (e.g.: IEEE 830-1998), which should be maintained by a RET. Need for separation between requirements analysis and requirements specification (validates the feature FEF3.2). analysis should be followed with different reports, agreement, negotiation, and documentation. specification is the activity of documenting a software requirements specification. The FEF3.2 is one of the most important features, as the quantitative analysis has showed. Need for requirements grouping (validates the feature FEF2.2). The project has to describe requirements for time constraints, functionality, usability, reliability, information storage, source code. The FEF2.2 is quite well supported by the RETs candidates (RET2, RET4, RET5, RET6, RET7) but the tools have many shortcoming concerning different modeling perspectives and participant viewpoints. Need to specify using different representation techniques, including informal, semiformal, formal specifications. (validates the features FEF1.1, FEF1.2, FEF1.3). The requirements model includes one logical unit, while the variety of project participants demands for different representation techniques for various requirements groups during all the RE process. In the project, two techniques were used for informal requirements representation natural language and use case templates [17]. Semiformal representation included state transition diagrams and reference modeling language [29]. In order to represent dynamic relationships between requirements, requirements model was extended by formal set theory description. Lack of support for multiple, distributed users (validates the feature FEF2.3). The RET should include multidisciplinary team work, which could be geographically distributed. The FEF2.3 is not supported by the RETs candidates or it is supported quite weak. Need to provide means for communication, maintenance of rationale (validates the feature FEF2.1). It is important to be able to recreate the rationale behind some requirements specification items [21]. It was quite a challenging task in the project. First, because of the different communication tools (e-mail, MSN and ICQ messenger programs) it is difficult to support and argue appropriately the different issues of requirements. Second, the rationale needs be related to each element of requirements specification. Need for maintaining traceability relationship among different requirement elements (validates the feature FEF1.4). Traceability is needed to relate requirements, their rationale, source, requirements representation and requirements specification versions. Some of the RETs candidates (RET1, RET7, RET8) provide adequate support for the FEF1.4. 8

Repository needs for storing data about requirements specification (validates the feature FEF3.1). A RET should support storage of requirements in a requirements repository instead of a paper document. The requirements repository stores requirements related information such as: individual requirements, requirements metadata, different requirements representations, and requirements models. It would benefit to set traceable relations between various elements of the requirements specification. The repository should provide version control and reuse of already agreed common domain requirements. Support different data formats according to modeling techniques and tools used (validates the feature FEF1.5). A tool should allow export and import of requirements models. This would benefit the means to specify requirements using different paradigms and various modeling techniques. Support for flexible requirements management. The study [23] shows that RE process should not be characterized by a smooth and incremental evolution, but by occasional crisis points where the requirements models are reconceptualised and simplified. A RET needs to promotes design creativity and support reconceptualization of the requirement model for restructuring the requirements specification. The maintenance of requirements specification shows the important RE aspects, which are missing while using editing, drawing and modeling tools. The executions phase demonstrates validation issues for evaluation framework [22]. We can easily notice that the framework covers the shortcomings, which are needed during the requirements specification maintenance. 5. CONCLUSIONS The paper analyses the evaluation framework [22] for RETs. The adaptability and usability tests help to find out the important characteristics of an environment and the needs for a RET acquisition. The observation of the maintenance requirements specification using editing, drawing and modeling tools highlights the requirements for the proper automatic support of the RE process. The results of both analysis the quantitative analysis and observation contribute towards validation of the evaluation framework for RETs. 5.1. Discussion The usability and adaptability of the evaluation framework [22] is one of the key issues during the evaluation of RETs. The important needs of stakeholders, who are going to work with a RET, are explained. Further, the adaptability and usability studies show the validity. The majority of the participants include researchers, with different computer science backgrounds and experiences. First, the participants describe the organizational reality from their own perspectives. Second, the different educational background is the potential threat that could affect the interpretation and understandability of the questions and reliability of the answer. In order to maintain the unique interpretation of the questions, the discussion about the project was held, where the project objectives were presented. Participants experience and knowledge could produce a big variance of agreement for framework features during the quantitative analysis. The problem solution could be the setting of flexible threshold in order to remove the most non-agreed features but still leaving a sufficient list of features. After the evaluation framework highlights the most important environmental needs, it becomes possible to describe RETs and to express the quality requirements and needs of an environment. It is important to recognize that the evaluation of the RETs candidates relies on subjective opinions of evaluators. But the proposed techniques for the framework acquisition contribute towards the objective evaluation method, because different organizational representatives are involved during the framework acquisition and the RETs candidate evaluation. The collected information should be reused for the other RETs evaluation phases, where tool candidates could be tested with practical engineering examples. The quantitative analysis provided a useful knowledge for maintenance of requirements specification, using editing, drawing and modeling tools. It showed the critical requirements, needed for the automated support of 9

RE processes. The results of the observation during execution phase contribute to validation of the evaluation framework [22] in high degree. 5.2. Future Work The evaluation framework [22] concentrates on functional requirements for RETs. In practice, most tool selections are not only effected by the functional attributes, but also by non-functional ones. The framework should be expanded with additional features and s for example non-functional enabling evaluation and validation of such activities as purchase and training costs, support, vendor reliability, reliability, usability, robustness, and stability. Adaptation of the framework to organizational needs depends on organizational profile, executives experience and organizational practice. We have applied the framework in an academic environment. It would be beneficial to explore the framework in industrial environment. Thus analysis and observation concerning RE practitioners should be carried out in order to investigate the framework [22] features from the industrial point of view. ACKNOWLEDGMENT The author would like to thank to Sari Hakkarainen and Guttorm Sindre for discussions and feedback concerning drafts of this paper. REFERENCES [1] Botella, P.; Burgués, X.; Carvallo, J.P.; Franch, X.; Quer, C., Using Quality Models for Assessing COTS Selection, Workshop on Engineering, WER 2002, València [2] Caliber-RM: URL: http://www.starbase.com/ [3] CORE: A Guided Tour. Release 3.0. 12.2000, URL: http://www.vtcorp.com/productline.html [4] Cradle: Cradle User Guide & Tutorial, URL: http://www.threesl.com/ [5] Davis A.M, Overmeyer S., Jordan K., Caruso., Dandashi F., Dinh A., Kincaid A., Ledeboer G., Reynolds G., Sitaram P., Ta P., Theofanos A., Identifying and measuring quality in a software requirements specification. In proceedings of the First International Software Metrics Symposium pages, 141-152. [6] DOORS: Using DOORS, 12.06.2001, URL: http://www.telelogic.com/ [7] Ferdinandi, P., L., A Pattern. Succeeding in the Internet Economy, Addison-Wesley, 2002 [8] Franch X., Carvallo I., A Quality-Model-Based Approach for Describing and Evaluating Software Packages, in proceedings of IEEE Joint International Conference on Engineering (RE 02), Essen, Germany, 2002. [9] Harrison W., Ossher H., Tarr P.: Software engineering tools and environments: a roadmap, The Future of Software Engineering, Anthony Finkelstein (Ed.), ACM Press 2000. [10] INCOSE: Tools Survey: Management (RM) Tools by International Council on Systems Engineering (INCOSE) URL: http://www.incose.org/tools/tooltax.html [11] Kaindl H., Brinkkemper S., Bubenko Jr. J., Farbey B., Greenspan S. J., Heitmeyer C. L., do Prado Leite J. C. S., Mead N. R., Mylopoulos J., Siddiqi, Engineering and Technology Transfer: Obstacles, Incentives, and Improvement Agenda, Engineering (2000), 7: 113-123. [12] Karlsson L. Dahlstedt A.G., Natt och Dag J., Regnell B., Persson A., Challenges in Market-Driven Engineering - an Industrial Interview Study in: proceedings of REFSQ'2002. Essen, German( 2002) [13] Kelly S., Lyytinen K., Rossi M., MetaEdit+ A Fully Configurable Multi-User and Multi-Tool CASE and CAME Environment, Proc CaiSE 96, Heraklion, Hellas, May 1996 [14] Kotonya G., Sommerville I., Engineering: Process and Techniques, Wiley, 1998 [15] Krogstie J., A Semiotic approach to Quality in Specifications, in proceedings IFIP 8.1 Working Conference on Onganizational Semiotics, Montreal, Canada, 2001. [16] Krogstie J.: Integrating the understanding of Quality in requirements Specification and Conceptual Modeling. Software Engineering Notes 23 (1), (1998) 89-91 [17] Kulak D., Guiney E., Use Cases: in Context, Addison-Weslay. 10

[18] LaBudde Ed. V.: Finding the Right Off-the-Shelf Management Tool, (MDDI, Oct 1997, p. 48). URL: http://www.devicelink.com/mddi/archive/97/10/013.html (checked: 19.05.2003) [19] Lang M., Duggan J.: A Tool to Support Collaborative Software Management, Requirement Engineering 6 (2001) 161-172 [20] Lindland O.I., Sindre G., Sølvberg A.: Understanding Quality in Conceptual Modelling, IEEE Software 11, 2 (1994) 42-49. [21] Loucopoulos P., Karakostas V., System Engineering, McGraw-Hill, 1995. [22] Matulevicius R., Strašunskas D., Evaluation Framework of Engineering Tools for Verification and Validation, in proceedings of the International Workshop on Conceptual Modeling Quality (IWCMQ 02), Tampere, Finland, 2002 [23] Nguyen L., Swatman P. A.: Managing the Engineering Process, in proceedings of REFSQ'2001. Interlaken, Switzerland (2001) [24] Nikiforova O., Sukovskis U. Framework for comparison of System Modelling Tool, in proceedings of Fifth IEEE International Baltic Workshop On DB And IS BalticDB&IS'2002, Tallinn, Estonia, 2002 [25] Nuseibeh B., Easterbrook S.: Engineering: A Roadmap, The Future of Software Engineering, Anthony Finkelstein (Ed.), ACM Press 2000 [26] Pohl K.: The three s of requirements engineering: a framework and its applications, Information systems Vol. 19, No 3 (1994) 243-258. [27] RequisitePro: Rational RequisitePro v2002. Evaluators Guide with a Management Overview. URL: http://www.rational.com/ [28] RDT: Product Overview, URL: http://www.igatech.com/rdt/ [29] Sølvberg A., Introduction to Concept Modeling for Information Systems, Department of Computer and information sciences, Norwegian University of Science and Technology, IDI - NTNU. January 2000. [30] VitalLink: Vital Link Tutorial & Help, URL: http://www.complianceautomation.com/ [31] Wiegers K. E.: Automating Management. URL: http://www.processimpact.com/articles/rm_tools.html (checked: 19.05.2003) [32] XTie-RT: Cross Tie, Version 3.1.03 Tutorial, URL: http://www.tbe.com/ [33] Zave P., Classification of research efforts in requirements engineering, ACM Computing Surveys, December 1997 11