Simulation Tactics, Strategy and Interpretation G A Vignaux August 29, 2004 1
Contents 1 Introduction 3 2 Developing Simulations 5 3 Traces 9 4 Terminating versus non-terminating simulations 15 5 Output Analysis 23 6 Variance reduction Methods 29 7 Other Simulation Topics 36 2
1 Introduction A simulation must be written to answer a question, not as a general toy. Ask what are the (specific) effects of doing something. For example: What is the change in average delay time if we increase the number of servers in the system by 1? You must interpret the output with care. Consider the risks involved in a wrong decision. A simulation program must be verified (does it correctly replicate the designed model?) and validated (does it correctly replicate the real situation?) 3
1.1 Statistical nature of simulation Runs of the simulation are statistical experiments and must be analysed as such. We must be able to distinguish between (1) variations due to changes in the system configuration and (2) sampling variation in the results. Serial correlation in observations give us problems Control over random variables is a great help. 4
2 Developing Simulations Design, Language, developing code, testing, validation. 2.1 Design Initial Design of the model. What questions are to be answered? Modelling with ACDs (2.1.1). 5
2.1.1 ACD 6
2.2 Simulation Language Choice of a simulation languages. Advantages and limitations of different approaches. Compiled: Simula, Simscript, C-sim, C++-sim Scripted: SimPy Packages: SIMON, HOCUS, Graphical: Simul8 7
2.3 Developing code Develop in an incremental manner. DO NOT write all the code out at once. Use Deterministic times at first, NOT random ones. 2.4 Testing Testing, Traces (see Section 3). Unit tests. How do you test a system with random numbers? If you have control over RVs you can find out what they should be. What is the difference between validation and verification? 8
3 Traces Trace files are essential for developing, debugging (Section 3.2), and running a simulation. It is a text file generated by the simulation. Consists of a series of lines. Each line records an event. Events: instants when the state of the model changes A trace line consists of: simulation time always first Followed by any other information about the event This format is chosen so some, now unknown, program can read and analyse the trace later. (eg S-Plus, R, ) 9
Example 3.1 10.0325 Customer 3 arrives 11.0567 Customer 2 departs 11.0567 Customer 2 leaves the system 11.0567 Customer 3 starts service at node 5 12.0033 Customer 3 departs 12.0033 Customer 3 transfers to node 4 The first element on a line is the time. It is possible to work out when customer 3 arrived, went into service, and departed. 10
3.1 Producing a trace Either by print statements or by aspecial version of the simulation package. The simplest way of producing a trace is to include conditional trace statements in the code at suitable points. These can be print statements or calls to a trace method. 11
Example 3.2 Here is an example of a trace method for a SimPy Customer process: def trace(self,message): if tracing: print %7.4f pax %03d %s %(now(), self.id, message) If the boolean tracing was set to True (or 1), a statement like self.trace( arrived ) would print a trace line with the simulation time, the customer s identification number, and the message arrived. 12
3.2 Traces for debugging Simulation programs are notoriously difficult to get running properly. They involve running artificial parallel interacting processes - rather like simple operating systems. The particular complication is the random numbers built into most simulations. We always build-in the ability to generate a trace file while running the program. We should be able to switch it on or off by setting a parameter for the run so that the trace does not interfere with the results of the run. Using this technique it is usually easy to see any flaws in the logic of the model. If, as we should, we start by using non-random variables when we develop the model this logical verification is much easier with a trace. 13
3.3 Traces for output Traces are often use as an intermediate file between the simulation and analysis of the output. The trace records a complete run of the model. It it is written so it is easy for another program to read in, much of the statistical analysis and presentation can be carried out by programs designed to do just that. For example the trace output of a Simscript model might be read in by an S-PLUS script to find serial correlations. It may be useful to run the trace through another script (in Perl, Python or shell script) to massage the data before sending it to the statistical package. 14
4 Terminating versus non-terminating simulations There are two general classes of system simulation models: 4.1 Terminating Some models terminate naturally, such as a shop closing its doors at the end of the day. To get more accuracy runs have to be replicated. In many cases we will be interested in transient measurements: such as those where we want to measure the starting effects. For example, the performance of the airport system when a 747 arrives at Auckland. 15
4.2 Non-Terminating Non-terminating Some models have no natural termination, such as a 24-hr bus terminal. To get more accuracy, runs have to be increased in length. We would probably be interested in making steady state measurements: such as those needed to compare with theoretical calculations in queue or queue network theory. This leads to some tricky decisions in running the simulations. 16
4.2.1 Achieving steady state for non-terminating simulations y 40 Number of restaurants 35 30 25 20 15 10 5 00 2 4 6 8 10 12 14 16 18 20 t For non-terminating simulations we want to measure operations in steady-state. 17
The easiest way to start a simulation is to have no activity occurring and all queues empty. ( Empty and Idle ). But as this state is not typical, this can bias the results. Two ways out of this difficulty are to use a run-in( or warm-up) period or to start at a typical state. 18
4.2.2 Using a warm-up period y 20 Price of Pizza 18 16 14 12 10 8 6 4 2 00 2 4 6 8 10 12 14 16 18 20 t Reject the results for the warm-up period. The simulation proper begins at the end of the warm-up. Statistics are collected from then on. 19
4.2.3 How long should the warm-up period be? There seems to be no sure statistical test for this. One way is to monitor statistics through an entire run and check for the time steady state is reached. Nowadays this can be done using graphical output to check, for example, that average queue lengths are not gradually increasing any more. Otherwise graphs of trace output may suffice. Another is to determining the run-in period in advance and use the same run-in period for all the experiments. This run-in period may be determined by hunch, from an initial simulation or by analysis of a simplified model. 20
4.2.4 Starting in a typical state y 40 Number of restaurants 35 30 25 20 15 100 20 40 60 80 100 t An alternative is to begin each run of the simulation with non-empty conditions that are somehow typical. 21
These conditions may be quite complicated. They might be determined from knowledge of the system, from an analysis of the ACD, or from the results of preliminary simulations. This may be difficult to set up; e.g. if you spread entities round the system, all waiting to start the next activity how do you get response times for those initial ones? 22
5 Output Analysis Simple statistics (in the Simulation language). SimPy Monitors. Graphical Output (5.1) Simple Comparison of Runs Statistical Analysis of Results (5.2) 23
5.1 Graphical Output Simple types of output, eg graphics. Graphs, Histograms. y 600 Number of restaurants 550 500 450 400 350 300 250 200 150 100 50 0 24 26 28 30 32 34 x Example. 24
5.2 Statistical Analysis Estimation is simplest when the observations are statistically independent and identically distributed. Then the central limit theorem, implying Normal distributions for totals and hence means, can be used to estimate ranges of uncertainty or confidence intervals. But in many simulations, observations of a single variable are likely to be autocorrelated. Then, even though the estimates are not biased our estimates of uncertainty are more difficult. Measure the Correlation and correct for it Replication Batching Regeneration 25
5.2.1 Simple Replication The simulation is repeated several times with the same conditions except that different streams of random number are used for each run. This ensures that the replications are independent of each other. Thus for a random variable X resulting from n replications of the same simulation, the mean value is the overall mean of the n replications and the overall variance is the mean of the n variances. 26
5.2.2 Batching A single long run is divided into batches of a given number of completions or a fixed time. The measurements made in the batches will be correlated but not very much if the batches are long. The batches are then regarded as replications. 27
5.2.3 Regeneration A single long simulation run is broken into batches by dividing the run at regeneration points. These are identical system states where the system appears to start anew in a statistical sense. Thus an M/M/1 queue would have a regeneration point whenever a job leaves it empty. Although the batches are not the same length, the method guarantees that there is no correlation between them. The analysis is described in [1, Ch 23.4]. 28
6 Variance reduction Methods These are methods to increase the precision of the response variable estimates. They are often called Monte Carlo methods. They were invented primarily for use in statistical methods of estimating integrals and can be particularly effective there. Results in simulating real discrete-event systems are sometimes disappointing. See [1, Ch 23]. Some methods include (6.1) Common Random Numbers, (6.2) Control Variates, (6.3) Antithetic Variates, and ((6.4) Importance Sampling. 29
6.1 Common random numbers This takes advantage of the control we have over the streasm of random numbers used in the simulation. One can hold the sampling variation constant across the alternative policies by using the same random numbers in each run. The differences between results is then (mainly) due to the different policies. Each source of variation in the model can have its own stream of random numbers. For example, the random variates used to generate inter-arrival times, the CPU service time, and the choice of paths would all be from different independent streams. 30
In SimPy, this would be done using different random variable objects for each source of variation. For different runs the seeds for each stream are reset to the same initial values. Then, if synchronisation of the random numbers is achieved, each entity retains the same values of random variables for corresponding actions in all the runs and behaves in a similar way. We hope that sampling variation will be minimised. There is positive correlation between runs. The variance of the differences between policies is reduced by the covariance between the corresponding runs. 31
6.2 Control Variates Additional observations are made of variables that might be expected to correlate with those being measured. For example, if we are interested in the average response times in a queue and measure W, we might also observe the average service times we generate, S. If a service time is high we would expect the corresponding response time to be high. We also know what the true mean of these service times, T s, should be (since we are generating them from the service time distribution). 32
We correct the observed average response time by (some proportion of) the difference between the actual and expected service time means, W = W λ(s T s ), where W is the corrected mean response time. The value of λ can be established by regression measurements in pilot simulation runs. Note that no matter what value of λ is chosen, the estimate of W will remain unbiassed but its variance may be increased if you choose badly. 33
6.3 Antithetic Variates Each run is duplicated. The second run in a pair uses an identical random number stream to the first BUT corresponding RVs are negatively correlated. This is usually done by replacing each PRN U i (uniform 0 to 1) by its antithetic value 1 U i. Then a large service time in the first run will have a correspondingly small service time in the antithetic run. Thus the two runs should have resulting measurements X 1 and X 2 that are negatively correlated to some degree. We use the mean of these, (X 1 + X 2 )/2 as the resulting measurement. This should have less sampling variance than either of the two or even of a run the same length of the two put together. 34
6.4 Importance Sampling If we are interested in system states that occur very rarely (eg rare failures) we may distort the probabilities so as to increase the chance of this happening in our simulation. We then adjust the results for this distortion. This requires a relatively simple situation so we can produce a probability model so the effect of the distortion can be allowed for.. Dr Peter Smith (ex-mcs, now at Canterbury) is using this technique to study the effect of error detection and correction methods in Telecom transmissions. In these studies bit error rates of 1 in 10 11 are typical. There is no way to simulate this without the leverage of importance sampling. 35
7 Other Simulation Topics Stochastic Optimisation Methods Genetic Algorithms Simulated Annealing Gibbs Sampling 36
References [1] Frederick S Hillier and Gerald J. Lieberman. Introduction to Operations Research. McGraw-Hill, 5th edition, 1990. [2] Averill M Law and W D Kelton. Simulation Modeling and Analysis. McGraw-Hill, 1982. [3] Michael Pidd. Computer simulation in management science. John Wiley & sons Ltd, Chichester, 2nd edition, 1988. [4] Michael Pidd, editor. Computer Modelling for discrete simulation. Wiley, 1989. [5] Wayne L Winston. Operations Research, Applications and Algorithms. Duxbury Press, 4th edition, 2004. 37