Practical experience with the application of HazOp to a software intensive system

Practical experience with the application of HazOp to a software intensive system Tor Stålhane, PhD and Kari Juul Wedde. M Sc. SINTEF Telecom and Informatics Abstract This paper describes the work done by SINTEF on HazOp on a safety critical, software intensive system and the lessons learned in the process. The lessons concern two areas the HazOp process and the use of the HazOp to formulate lower level safety requirements. We describe a HazOp process based on system functions instead of the overall system functionality and how to use software related guidewords in order to fill in the HazOp forms. For requirements generation and allocation, we describe the process of construction a FT based on the contents of the HazOp forms and how to perform the probability budgeting. Keywords HazOp, Fault Tree Analyses (FTA), System safety analysis, Software intensive systems 1. Introduction SINTEF has for some years performed safety analyses for European companies working in the areas of rail and air traffic control. One of our business ideas is to use traditional safety analyses techniques to analyze software systems. If at all possible we want to use these techniques unchanged but if this proves impractical we will augment them so that they can also be used for software. One of the safety analyses techniques where we saw a great potential was the HazOp Hazard and Operational analyses. This technique can be performed early in the development process - preliminary HazOp - or when the design is (nearly) complete - a regular HazOp. We have used both in this project. Many of the analyses we have performed earlier were for systems that were already finished and our task was to answer the question Is it safe enough? For such work HazOp is only needed in order to get an overview of the potential dangers. For the project discussed in this paper, we participated already from the concept phase and the HazOp result was one of the main forces driving the architectural design. This paper sums up our experiences on safety analyses over several years. This experiences has been increased and validated through the work in the project described below. 2. A short summary of state of the art HazOp as a method dated back to the 60s and was developed by ICI [1] for the analyses of chemical processes. The method, with its focus on the early stages of development did, however, soon catch the attention of other branches of industry where safety is an important issue. The software industry was probably the last industry to put it into use. In 1986, Nancy Leveson recommended the method for analyses of all safety critical software systems [2]. The problems pertaining to the fact that the method had been designed for chemical processes was, however, not mentioned. The same holds for EWICS treatment of the subject in 1988 [3] and for Lewis Bass paper in 1989 [4]. The first attempt to tailor HazOp to software was reported in 1989 [5]. The authors Dr. R.M. Pitblado et al recommended a standard HazOp to be performed first, followed by what the authors called a failure mode HazOp, where the authors used many of the standard

guidewords, but combined them with new, software adapted parameters. The first really software adapted HazOp description was published by Chudleigh and Catmur in 1992 [6]. The authors introduced new guidewords and new parameters. At the time of the publication, they had already performed several software HazOps but had been unable to publish the experiences due to clauses of confidentiality. When we sat down to make a new, better adopted, set of guidewords for software HazOp, the work of Chudleigh and Catmur was our main source of inspiration together with work done at ESTEC [7]. 3. System and related problems Figure 1 shows the system and its environment. The main purpose of the system is to generate and broadcast position information based on a set of input signals and geographical position information. For safety reasons all messages have to be monitored before and after sending. Mismatch in message contents shall cause an alarm to be sent to the receiver and if the failure conditions continuous; it shall lead to a system shutdown. Status information shall be displayed - both on a local and on a remote panel. The geographical positions are loaded and stored into the system when the system is set to maintenance mode. Data are not allowed to be entered during normal operation. Changing the operational mode is done by an operator. Receiver Sender Messages for monitoring System Safety critical part Operator Geographical positions Remote Panel Figure 1 The system The only unsafe event for this system is that misleading information is sent to the receiving system. Loss of function is for instance not a hazard. The critical functions are thus

as follows: Broadcast position messages and alarms Monitoring of the broadcasted messages. The ability to reveal incorrect input signals. The ability to reveal incorrect geographical data. Another interesting aspect is the role of the operator; what harm can the operator cause by applying the wrong command, pushing the wrong switch and so on. As can be seen from Figure 1, all critical functions are related to the interfaces between the safety critical part and its environment. The receiver and the senders are not part of this project and thus not part of the HazOp. For the parts inside the dashed line, however, the interface between safety critical and non-safety critical parts have to be considered thoroughly. 4. HazOp a method description 4.1. What is HazOp? As an idea, HazOp is simple - collect a group of people, ask them to identify potential hazards linked to the system and document the result. HazOp is a guided and structured brain storming process. The guidance is provided by the guidewords while the system or the system s functions provide the structure. HazOp is a people process and shares many of the standard characteristics of other group processes, first and foremost the fact that nothing comes out of a HazOp that is not first present in the mind of the participants. The method is used to get an overview of all potential hazards. It can be used in two ways, namely: For safety assessment, where we check that all potential hazards can be handled by the system. For system design, where we start with a preliminary HazOp and then enters into a loop consisting of design, HazOp and fault tree analyses in order to check whether the goals can be reached. This paper will concentrate on the use of HazOp in system development. 4.2. How to do a HazOp HazOp is a general technique and is just as important for hardware, software and peopleware. The following are the most important things to remember when performing a HazOp: 1. HazOp is a group process. The quality of the HazOp result is a direct function of the quality of the personnel involved. The group must contain experts on all relevant areas software, hardware, application, installation, maintenance and operation. Only in this way can we cover all potential hazards. See also [8]. 2. As a process, HazOp has many features in common with a document review and many of the same rules should be applied. First and foremost, the group s role is to identify hazards, not to come up with design solutions. 0It is important to draw clear system s boundaries so that the participants always know what is inside the system and what is outside. 1. All modes of operations must be considered. Include also installation, maintenance and operator interventions. HazOp has two important features the study nodes and the guidewords. We will give a short discussion of the role of guidewords and study nodes in the next section. The results are documented in a HazOp form. An example of such a form with some typical information is shown in figure 4.

The guidewords drive the HazOp process by focusing the brain storming on a set of deviations from normal operation, which can occur during operation or maintenance. In some sense the guidewords can be viewed as generalized failure modes. The guidewords used in a HazOp performed in the chemical industry are for instance No negation of the design intent, Less quantitative decrease or More quantitative increase. The guidewords are combined with a parameter for instance flow - and a study node for instance valve V3. A possible failure mode can thus be No flow in valve V3. Such guidewords are not particularly well fitted for software. There has been some work done on interpreting them for software, see for instance [5], but the idea has not gained wide acceptance. 5. From hazards to HazOp forms 5.1. Some problems pertaining to HazOp Study nodes The first important choice when starting a HazOp process is the choice of study nodes. This choice will decide the result of the HazOp and how the result can be used. We have two possible choices for study nodes, namely system components (channels, processes and data storage) or functionality. The latter will focus on what the system delivers to its environment, for instance the signals sent to the actuators. If we choose to perform a functional HazOp this can be done in two ways: 1. Define the study nodes of the system and apply the guidewords to each function available in the current study node. 2. Start with the functions defined by the system s operation and apply the guidewords to each function or sub-function. The first alternative is the method recommended for HazOp in other industries. It turned out, however, that we at least for software - got a more efficient process by selecting the second alternative. Especially, it turned out that the iterations between HazOp and fault tree analyses became much easier in this way. We identify all functions for the current system level and applied the guidewords belonging to process, data storage and channels, together with their deviations to each function. Guidewords As mentioned earlier, attempts to interpret the original HazOp guidewords in a software setting, has not been much used and we have chosen another approach. In 1995, SINTEF headed an ITUF working group with the goal of defining special guidewords for software systems. The work resulted in three sets of guidewords, one for each of the study node categories Process, Channel and Data storage. Part of the results is shown below: Table 1 Guidewords Parameter Storage Input / output Data value Guideword Unstable Destroyed Stored in wrong address Stored in such a way that it can not be read Noise or other random input is stored as data Total or partly wrong Unwanted data No data value Not complete data Data are not changed

The guidewords developed by ITUF were originally intended for system components. The standard we used [10] in this particular project, however, required that we focused on functions. It turned out that they also worked quite well in this case. Filling in the HazOp form The HazOp forms, which are the output of the process, are filled in by going through the list of guidewords for each system function. Thus, the parameters - which originally were components - are now augmented with the functions. For each function we run through all the guidewords plus earlier identified hazards, to check whether they are applicable to the current function and parameter. Thus, instead of using the failure unstable storage, we will use the failure unstable storage for function F3 and so on. If the failure is relevant, the condition under which it can occur is written into the Failure Condition column in the HazOp form. These conditions can be: Single conditions, such as Bit error in status word. Multiple conditions, such as Bit error in status word and not detected by CRC. For each failure we must also fill inn the effect, classification and how we shall verify that the hazard is properly taken care of later in the process. 5.2. Lessons learned HazOp must be done by iterations. It is not possible to do a complete HazOp in a single run through all the nodes or functions of a system. Hazards identified in one function may turn out to be relevant also for other functions, though not necessarily in the same way as the first time. Thus, when doing a HazOp and identifying a potential hazard, we must go back and check if this hazard or something related to this hazard also applies for the other functions. In addition, the hazard must be added to the list of potential hazards we need to consider for the functions that are not yet analyzed. Fault scenarios and event sequences are important. If we are performing a HazOp on a complex system it is important to use What if scenarios. This will help to identify the functionality involved in a potential hazard. Event sequences are closely related to fault scenarios and serve the same purpose. Thus, instead of just considering the failure bad data in X, we should consider the sequence of events that leads to the bad data. This will help us to create more efficient defense against the identified problem. Any solution introduced in order to get rid of a risk may introduce a new risk. Any change in architecture or design, intended to solve an identified problem, can introduce a new hazard. A typical example is a new control function intended to remove a hazard. By introducing this new function, we create new hazards and so on. This insight will affect us in the future mainly by the fact that we are aware of the problem. In addition, introducing new control mechanisms may not be a good way to get rid of a problem. It is better to remove the potential danger by changing the design than to introduce extra control mechanisms. See also the previous lesson. System HazOp versus functional HazOp. As mentioned earlier, a functional HazOp can be done either by starting with the job that the system is supposed to perform or by starting with the functions that are used. Our experience is that it helps the rest of the process tremendously if we start with defining the main functions of the system and then perform a HazOp on each function. This will simplify the rest of the safety analyses e.g. the FTA and the mapping from

HazOp to system s design. The result will be easier to document, understand and review. Define system boundaries It is important to have a clear description of the system s boundaries. We found that it is also important to have a clear description of which inner parts of the system are visible at the time of the HazOp. If this is not made clear we will get into trouble when we try to separate HazOp and design work. 6. From HazOp forms to design requirements 6.1. The role of HazOp in architectural design One purpose of HazOp in architectural design for safety critical systems is to perform quantitative analyses. Such analyses are done in order to get a starting point for the generation and allocation of lower level safety requirements. This process is iterative in nature and consists of FTA, probability budgeting and design modifications. HazOp applied to architectural design differs from HazOp applied on existing systems in two ways: The probability budget The iterations between lower level design and the HazOp process When applying FTA it is important to define the analysis level required. For architectural design, we constructed a FT down to the level of events representing single failure conditions in the HazOp forms. The reason for going down to single failure conditions instead of stopping at multiple failure conditions was that some of the single failure conditions could be mapped into basic events with known failure rates. In this way we could take advantage of extra knowledge and thus get a more accurate result in the probability budgeting. An important area related to probability budgeting is the lack of available failure rates for software and peopleware. This is a problem that is independent of whether the approach is applied during system development or on existing systems. Figure 2 shows a part of the HazOp form for the function Broadcast message and the FT developed based on this form. The FT is developed down to a level of single failure conditions. This example will be used throughout the rest of this chapter. 6.2. Probability budget When applying HazOp in architectural design, probability budgeting becomes an important part of the job. The probability budgeting is done by using FTA to allocate failure rates to the failure conditions identified in the HazOp form. In probability budgeting you start with the safety requirement for the top event. Normally you divide the failure rate equally between all events at the level below. For an OR-gate with two events, each event will get a failure rate that is half the failure rate of the starting event. For AND-gates with two events, each event will get a failure rate that is the square root of the failure rate of the starting event. This simple top down approach to budgeting can give good results, but requires that the functions are grouped in such a way that it is reasonable to have the same requirement to each group. It is for instance unreasonable that a single bit error have the same failure rate as corrupted data. As an alternative to the top down approach you can use a bottom up approach. The FT is developed down to the analyses level decided upon, for instance single failure conditions. All basic events are given failure rates according to known rates and all undeveloped events are given equal rates which is adjusted until the top requirement is met. By applying this approach you can get an early indication of which subsystem or sub-function that may cause most problems.

6.3. Iterations between design and the HazOp process When the budgeting is finished, you have to formulate design requirements based on the undeveloped events in the FT and their failure rates. These requirements will serve as input to the design process. Here we make design decisions based on the requirements and develop FTs for the undeveloped events from the architectural design. In the design phase the FTs have to be developed down to basic events with known failure rates. If the top requirement is not met, you can alter the local design until the requirement is met or you can to go back and alter the architectural design. FunctionF ailure ConditionE ffect C lassificationve rification Broadcast message Bit error. Not detected by CRC.M isleading informationh azards ystem Fault TreeP osition data corrupted. Single failure. Not detected.m isleading informationh azards ystem Fault TreeP osition data corrupted. Multiple failure. Not detected.m isleading informationh azards ystem Fault TreeN o transmissionl oss of functionn o safety effect Figure 2 HazOp applied on architectural design 6.4. Failure rates for software and peopleware Allocation of failure rates to software and peopleware has always been a problem. For software this problem can be solved by deciding that software developed according to a given standard has a given failure rate. All software failures are then given the same failure rate. To get failure rates for i.e. operator failures are more difficult. There exists some numbers used in the area of nuclear energy [9] but the best advice is to design the system in such a way that the operator does not have any safety critical function. 6.5. The process of requirements allocation The starting point for requirement allocation is the overall safety requirements stated in the systems requirements and the HazOp form from architectural design.

1. Start a FT with the unsafe event for the overall system as the top event. The safety requirement for this event is taken from the systems requirements. In the system under consideration this unsafe event is Misleading information. 2. Construct an OR-gate connecting the main system functions. Allocate a budget for each function. Broadcast message in figure 2 is such a main function. 3. For each main function, construct an OR-gate connecting all failure conditions leading to the same effect. In order to get a more readable FT, related failure conditions can be grouped. In our example Position data corrupted is such a group. 4. If the failure condition is a multiple condition, the comprising single conditions are combined by an AND-gate, for instance Single data failure and Not detected. 5. Check for common causes. If this is not performed, the failure rate for the top event will be too high. In the example we have both single and multiple data failure where multiple failure includes single failure. The example does not compensate for common cause failures. 6. Allocate available failure rates to the basic events, i.e. Not detected by CRC. 7. Budget for the rest of the failure events and compute the failure rate for the top event. Iterate until the requirement is met. 8. If you are not able to meet the safety requirements, the system architecture have to be changed, and a new HazOp has to be performed. 1Formulate design requirements based on the undeveloped events in the FT. In the example, there will be three safety requirements. 6.6. Lessons learned Use a bottom up approach for failure probability budgeting We recommend using FTA and a bottom up approach to probability budgeting. In this way you can early in the architectural design process get an indication of which parts or functions in your system that will cause problems in meeting the overall safety requirements. Develop the FT down to the level for single failure conditions According to our experience, the analysis level required in architectural design is down to events representing single failure conditions. In the first version of our FT, we developed the FT as far as we were able to do with the knowledge available. Due to different level of knowledge in different areas, some hazards were developed down to a component level while others stopped at a high level. If we had used this FT as a basis for formulating design requirements, the requirements would be on quite different levels of abstraction and not well suited for design decisions. Iterate between HazOp and FTA. It can not be expected that the overall system safety requirement is met in the first FTA. You have two alternatives: 2Change the system architecture in such a way that one or more hazards will disappear. 3Reduce the failure probability by adding controlling or detection mechanisms to the system. The HazOp form and FTs can be used as a guide in order to choose the mechanism having the most effect on the overall failure probability. One to one mapping between elements in the HazOp form and FT event If the contents of the HazOp forms have a good structure, the FT can be constructed by a one to one mapping between elements in the HazOp form and events in the FT. 4Effect top event for all failure conditions leading to this effect 5Multiple failure condition grouping event for the comprising single events 6Single failure condition basic or undeveloped event

One design requirement for each undeveloped event in the FT Formulate one design requirement - allocated to a function or item - for each undeveloped event in the FT. At this level a SW failure is a SW failure At the architectural level we are not able to distinguish between the failure rates for different software failures. Even if we know that several failure conditions have to occur, we can not connect them with OR-gates or AND-gates the way we do for other types of events. Using the given standard for development of safety critical software, the failure rate for software is given. The following computing rules for SW failures must be applied: 7SW failure rate AND SW failure rate = SW failure rate 8SW failure rate OR SW failure rate = SW failure rate 7. Conclusions and further work We have seen how HazOp can be an integrated part of the design process in order to formulate and allocate design safety requirements for the lower levels. Pertaining to architectural design of software intensive systems, the following are important: 9The HazOp process must be iterative. This goes both for filling in the HazOp forms and for iteration between the HazOp process in the architectural design and the lower level design. 10It is more efficient to perform HazOp on a set of defined functions than to start out with the overall functionality. 11Applying HazOp in design includes probability budgeting. In order to perform this task in an efficient manner, failure rates for software and peopleware are required. HazOp as a method is well suited for software intensive systems. In order to be more suited for the design process, however, further work is needed in the following areas: 12List of generic hazards, according to application area 13Standard failure rates for peopleware and software failures 14SINTEF will try alone or together with European partners to initiate research in one or more of these areas. 8. References [1] A Guide to Hazard and Operability Studies. Chemical Industries Association Ltd. 1987. [2] Nancy G. Leveson. Software safety: Why, What and How. Computing Survey. Vol. 18, no. 2, June 1986 [3] EWICS TC7, Safety - Assessment and Design of Industrial Computer Systems. Techniques Directory, August 1988 [4] Lewis bass and Daniel L. Martin, Cost-Effective Software Safety Analyses. Proceedings of the Annual reliability and maintainability Symposium, Atlanta, Georgia, January 24-26, 1989. [5] R.M. Pitblado et al. safety Assessment of Computer Controlled Process Plants. Proceedings of the 6th International Symposium on Loss Prevention and Safety Promotion in the Process Industries, Oslo, Norway, June 19-27, 1989. [6] M. Chudleigh and J. Catmur, Safety Assessment of Computer Systems using HazOp and Audit techniques. Proceedings of Safecomp 92, 1992. [7] Guidelines for Considering a Software Intensive System within FMECA Studies, ESTEC January, 1992 [8] Rito Kuivanen, Methodology for simulating robot system safety design. ISBN 951-38-4657-1. VTT, Espoo, Finland, 1995. [9] David I. Gertman and Harold S. Blackman, Human Reliability and Safety Analysis data Handbook. John Wiley & Sons, Inc. New York [10] SAE International, Guidelines and methods for conducting the safety assessment process on civil airborn equipment. SAE ARP 4761, December 1996