Proceedings of the 2001 Winter Simulation Conference B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, eds. VERIFYING AND VALIDATING A SIMULATION MODEL Anbin Hu Ye San Zicai Wang Simulation Research Center P. O. Box 126 Harbin Institute of Technology Harbin, Heilongjiang, 150001, P. R. CHINA ABSTRACT This paper presents the verification and validation (V&V) of simulation model with the emphasis on the possible modification. Based on the analysis, a new framework is proposed, and new terms are defined. An example is employed to demonstrate how the framework and terms related are used in verifying and validating an existing model. 1 INTRODUCTION Experimentation Black-box validation White-box validation Problem entity Data validation Black-box validation Comceptual modeling Simulation models are increasingly being used in problemsolving and to aid in decision-making. The developers and users of these models, the decision-makers using information derived from the results of the models, and people effected by decisions based on such models are all rightly concerned with whether a model and its results are correct. This concern is addressed through model verification and validation (Sargent 1991). The framework for simulation evaluation formed by problem entity, conceptual model and computer model blocks describing model assessment process as shown in Figure 1(Robinson 1997). The outer cycle along with data validity is the technical processes that must be addressed to show that a model is credible. Assessment activities are spawned from each of these technical processes. According to DoD5000.59, verification is the process of determining that a model implementation accurately represents the developer s conceptual description and specification ; validation is the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model (Defense Modeling and Simulation Office 1996). Problems in the framework and terms mentioned above are briefly accounted: Computer model Verification Conceptual Model Verification Figure 1: Simulation Modeling Verification and Validation Process Data of the real system lies in the center of the framework, and gaining the data is a basic requirement. So the framework cannot assess the simulation model when the real system is complex and the complete data is difficult to get, or when the real system and the simulation model are designed parallel. Analyzing code of the simulation mode to produce the documentation regarding the conceptual model and assessment processes is an assessment scheme for an existing model (Defense Modeling and Simulation Office 1996). However, It is difficult for an outstanding programmer to describe a conceptual model merely through analyzing code of the real system. Consequently, this assessment does not work. As it becomes obvious, the theory of V&V is valid in assessing a simulation model whose real system does exist because the data is vital in the framework. But the theory is not a good scheme to evaluate the existing model. So the theory and its framework have to be modified in order to broaden the range of assessing simulation model. 595
2 DEFINITIONS Figure 2 shows the modified framework composed of problem entity (), problem entity information (I), conceptual model (L), computer model () and computer model information (I). Hereafter, they are termed as five object (FO), and the modified framework is called as five object framework (FOF). The arrows refer to the procedures employed to verify and validate a simulation model. The intended uses are set at the center, in which is the real system or problem, I the adequate information to build conceptual model, L the theoretical model built by problem entity information to satisfy intended uses of the model, the computer program and implementation of the L, and I adequate information to produce the real system or answer the real problem. The FOF demonstrates that the process of developing a new simulation is to keep the balance of five objects. In the FOF, verification and validation need redefinition. I The Intend Uses Constraint Aim L I Destruction Weakness Figure 2: Assessment Technical Processes Verification is the process of determining whether each object satisfies the intended uses. Validation is the process of determining the balance of FOF from the perspective of the intended uses of the model. To use the framework to evaluate the model, five definitions are given below: Balance means that information of each object is adequate to satisfy the intended uses of the model. is the relationship of creation and promotion among the five objects. In other words, if A creates or promotes B, the relation between them is generation. A is B s mother and B is A s son. For example, in the FOF, generates I, is I s mother, and I is the s son. reveals that information can flow from one to another, which means son s information can be deduced from mother s information and the reverse is not the case. Mother s information is not necessary. Constraint is the relationship of confinement and restriction among the five objects. Regulation of constraint is that constrains I, I constrains, constrains L, L constrains I and I constrains. For instance, when building a simulation model, Modeling and Simulation (M&S) developers try their best to collect information of the real system or the problem. Some of the information will be found no use to build a from the perspective of the intended uses of the model, so the is used to cut fat of I, which shows that constrains I. and constraint indicate that the relationship between adjacent objects must be balanced, and strengthening or weakening any object will lead to imbalance. For instance, when conceptual model is simple, the information concerned is less than that in problem entity information. The balance is spoiled. Destruction is the relationship that one object constrains its grandchild too much to lead to normal constraints. If A generates B and B generates C, A is called C s grandmother and C is A s grandchild. Destruction gives rise to two cases: one is that an object is too strong to secure the balance. A good case in point is when building for a complex system, M&S developer cannot build a conceptual model to satisfy the intended uses with their knowledge because the real system is complex. The other is that an object is too weak to guarantee the balance. In considering the developing a simulation model, balance of the framework will never be destroyed if five objects have the same content. However, for a complex system, the conceptual model is weak in that the content of L is less than that of. The two cases will lead that destroys the L. Weakness is the relationship that an object is strong enough to constraint its grandmother. As in the case of destruction, weakness also involves two cases: one case is that an object is so weak as to inversely constrains its grandmother. For instance, L weakens when is too complex and M&S developers do not build a precise L with their knowledge. Another case is one object is weak to be constrained. For another instance, when computer model is too simple to guarantee the balance, L is Weaken by. Destruction and weakness indicate the relationship between two odd objects, and it is special generation and constraint when the balance among the five objects is destroyed. Destruction constrains an object in the same direction as constraint, however weakness constrains an object in inverse direction. Framework of Figure 1 is a special case in FOF. Destruction and weakness exist in a structure simultaneously. For example, in crystal growth process, some mechanism of crystal growth process is not clear, so it is impossible for the L and be high precise, in which destroys L and weakens. In a word, when destruction and weakness emerge in a structure, the contents 596
of information in FO are often found different and destructive to the balance. 3 APPLICATION L In the FOF, the intended uses of the model instead of data are at the center. Data is transformed into a part of the two objects: I and I and their factions prove to be selfevident. The contents in FO have to be the same so as to keep the balance. In FO framework, with generation, constraint, destruction and weakness, combining two objects can produce the other three. as shown in Figure 3.V&V Techniques, see Figure 4 (Balci 1997), proves that the FOF is in balance. FOF makes it easy to work out a better solution to assess existing model Full Scope and Training Simulation Model of Power Plant (Made in Russia) 210 MW Fiery Unit: Firstly, running to generate I, secondly analyzing I and to get I, Thirdly analyzing and to produce L, and finally using Verification and Validation Techniques in Figure 4 to prove FO to be in balanced in FOF. The whole process is shown in Figure 5. I b L I I 4 CONCLUSIONS I FOF emphasizes a balance of procedure (anything excessive or deficient will break the balance and exposes the disadvantages). M&S work must guarantee the FOF in balance and the V&V must prove the FOF in balance using the techniques and the information offered by M&S developers. FOF presents a reasonable scheme for the verification and validation of existing model and it also shows clearly that the conceptual model must be obtained from the combination of and, while other information can also be attained from it. FOF was used in verifying and validating the Full Scope and Training Simulation Model of Power Plant (Made in Russia) 210 MW Fiery Unit. I L c I d I L I I I L a e Constraint Destruction Weakness Figure 3: Two Object Produce the Other Three Process 597
REFERENCE Balci, Osman 1997. Verification, Validation and Accreditation of Simulation Models, Proceeding of 1997 Winter Simulation Conference, Edited by S. Andradottir, K. J. Healy, D.H. Withers, and B. L. Nelson. Defense Modeling and Simulation Office 1996. VV&A Recommended Practice Guide. Robinson, Stewart 1997. Simulation Model Verification and Validation: Increase the Uses Confidence, Proceedings of the 1997 Winter Simulation Conference, Edited by S. Andradottir, K. J. Healy, D.H. Withers, and B. L. Nelson. Sargent, Robert G. 1991. Simulation Model Verification and Validation, Proceeding of the 1991 Winter simulation Conference, Edited by Barry L. Nelson, W. David Kelton, Gordon M. Clark, pp. 37-47. Verification and Validation Inform al Static Dynamic Formal Audit Desk Checking Face Validation Inspections Reviews Turing Test Walkthroughs Cause-Effect Graphing Control Analysis Calling Structure Concurrent Process Control Flow State Transition Data Analysis Data Dependency Data Flow Fault/Failure Analysis Interface Analysis Model Interface User Interface Semantic Analysis Structural Analysis Symbolic Evaluation Syntax Analysis Traceability Assessment Acceptance Testing Alpha Testing Assertion Checking Beta Testing Bottom-Up Testing Comparison Testing Compliance Testing Authorization Performance Security Standards Debugging Execution Testing Monitoring Profiling Tracing Fault/Failure Insertion Testing Field Testing Functional (Black-Box) Testing Graphical Comparisons Interface Testing Data Model User Object-Flow Testing Partition Testing Predictive Validation Product Testing Regression Testing Sensitivity Analysis Special Input Testing Boundary Value Equivalence Partitioning Extreme Input Invalid Input Real-Time Input Self-Driven Input Stress Trace-Driven Input Statistical Techniques Structural (White-Box) Branch Condition Data Flow Loop Path Statement Submodel/Module Testing Symbolic Debugging Top-Down Testing Visualization/Animation Figure 4: Taxonomy of Verification and Validation Techniques 598 Induction Inference Logical Deduction Inductive Assertions Lambda Calculus Predicate Calculus Predicate Transformation Proof of Correctness
L Analyze Running Analyze I I I Proving FOF in balance Aim Move Constraint Weakness Destruction The Intend Uses L I V erification and V alidation Tech Figure 5: Process of Verification and Validation an Existing Model AUTHOR BIOGRAPHIES ANBIN HU received B.E in Electronic and Technology and ME in Flight Control, Guidance & Simulation from Harbin Institute of Technology in 1997 and 2000 respectively. He is now a Ph.D candidate majoring in Flight Simulator modeling method and simulation application. He obtained Meidi scholarship in 2000. His research interests include VV&A, Flight Simulator modeling method, fuzzy and neural network theory and application. His email address is <sanye@hope.hit.edu.cn>. YE SAN a Professor at Harbin Institute of Technology. He received his education at Harbin Institute of Technology. Prof. San has served his profession for more than fifteen years and has been awarded the Guanhua Fund Award and Excellent Young &Middle-age Experts of Heilongjiang Province. His research interests include system simulation and system control. His email address is <sanye@hope. hit.edu.cn>. ZICAI WANG a Professor at Harbin Institute of Technology. He received his education at Harbin Institute of Technology. Prof. Wang has served his profession for more than forty years. His research interests include system simulation and fuzzy and neural network theory. His email address is <sanye@hope.hit.edu.cn>. 599