INFORMS Transactions on Education

This article was downloaded by: [46.3.194.127] On: 11 February 2018, At: 07:08 Publisher: Institute for Operations Research and the Management Sciences (INFORMS) INFORMS is located in Maryland, USA INFORMS Transactions on Education Publication details, including instructions for authors and subscription information: http://pubsonline.informs.org An Interactive Spreadsheet-Based Tool to Support Teaching Design of Experiments S. T. Enns, To cite this article: S. T. Enns, (2008) An Interactive Spreadsheet-Based Tool to Support Teaching Design of Experiments. INFORMS Transactions on Education 8(2):55-64. https://doi.org/10.1287/ited.1080.0008 Full terms and conditions of use: http://pubsonline.informs.org/page/terms-and-conditions This article may be used only for the purposes of research, teaching, and/or private study. Commercial use or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher approval, unless otherwise noted. For more information, contact permissions@informs.org. The Publisher does not warrant or guarantee the article s accuracy, completeness, merchantability, fitness for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or support of claims made of that product, publication, or service. Copyright 2008, INFORMS Please scroll down for article it is on subsequent pages INFORMS is the largest professional society in the world for professionals in the fields of operations research, management science, and analytics. For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org

Vol. 8, No. 2, January 2008, pp. 55 64 issn 1532-0545 08 0802 0055 informs I N F O R M S Transactions on Education An Interactive Spreadsheet-Based Tool to Support Teaching Design of Experiments S. T. Enns Department of Mechanical and Manufacturing Engineering, University of Calgary, Calgary, Canada, T2N 1N4, enns@ucalgary.ca doi 10.1287/ited.1080.0008 2008 INFORMS This paper describes an interactive spreadsheet-based tool that can be used to generate data representative of the type that might be obtained running a structured set of experiments. The purpose of this tool is to help the user experience the iterative nature of design and analysis of experiments. The tool supports quick and simple generation of data for one and two-factor problems. The underlying relationships are based on queuing approximations for a single-stage batch production environment. Factor levels are related to product lot sizes and the response is assumed to be average lot flowtimes. Variability due to replication is emulated by sampling from a statistical distribution. Statistical software packages can be used to generate linear or quadratic models from the results generated. Analysis can include the examination of main and interaction effects or the optimization of lot sizes to minimize flowtimes. Key words: design of experiments, central composite design (CCD), response surface methods History: Received: July 5, 2005; accepted: January 23, 2006. This paper was with the authors 3 months for 2 revisions. 1. Introduction Experimentation and analysis is generally an iterative and interactive process in real life. This paper addresses the problem of teaching basic design of experiments (DOE) methodology from this perspective. In particular, a spreadsheet-based tool to generate data representative of that which might be obtained using structured experimentation is presented. Since the experimental design can be readily changed to generate new results, the tool supports following an iterative path in which the analysis of previous results is used to define further experimentation requirements. The most common approach in teaching DOE is to use textbook data sets. These are static and do not support capturing the true essence of experimental design and analysis as a multiple-stage process, with each stage being dependent on previous results. It is difficult to study the type of problem that requires a progression of steps. For example, the investigator may wish to the change values of the design points, alter the design type itself or simply run additional replications to increase confidence levels. As well, randomization of experiments is generally ignored when solving textbook problems since the actual experimentation component is missing. A logical alternative would be to have students run actual experiments. However, this approach also presents various challenges. First, it is likely to be very time consuming. Second, proper laboratory facilities must be available to run either physical or simulation experiments. Third, students may be overwhelmed by the process of running experiments, with attention being diverted away from the experimental design and analysis aspects of the exercise. Fourth, it is not that easy to design experimental scenarios that illustrate the intended behavior. For example, catapults are sometimes used to launch balls and the distance thrown is then measured as a response. Changing the catapult settings allows main and interaction effects to be observed. Replication is easy to perform but within-group variances are not likely to be equal at different factor settings, making analysis using analysis of variance (ANOVA) techniques inappropriate. As well, response surface problems where convex or concave behavior is exhibited may be hard to construct. However if resources can be committed to solving suitable problems, the approach of running actual experiments can provide a high level of experiential learning. A comprehensive example of such an exercise is given by Box and Liu (1999). This paper suggests a third approach, meant to support an iterative paradigm for teaching experimental design and analysis. Box (1999) provides a 55

56 INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS strong argument for the need to teach sequential investigation if statistical education is to show relevance to solving real world industrial problems. The idea here is to generate data quickly with the aid of a user-friendly spreadsheet-based tool. The development and use of such a tool is described in the remainder of the paper. It is meant to support the study of interactions effects, curvature and response surfaces for single or two-factor scenarios. Mock experiments can be quickly set up, evaluated and then altered for another round of design and analysis. 2. The Approach The basic idea behind this approach is to use a set of quantitative relationships in an underlying model that exhibit the desired type of behavior. These relationships should ideally exhibit linear or nonlinear behavior, depending on the factor settings. As well, they should support the demonstration of interaction effects. Furthermore, if the relationships are not well known or understood it makes the problem context more palatable with respect to the need for an experimental solution. The problem context implemented in this spreadsheet model is that of lot size selection in a batch production system. A big problem in manufacturing is establishing good lot (or batch) sizes for production. In batch production facilities it is common to have multiple part types processed on the same machine (or resource). These are capacity-constrained machines that can process only one part type at a time. It is common for each part type to have unique part processing time, lot size and lot setup time characteristics. The machine is typically set up for one particular part type and then a lot of parts is processed. The lot processing time is equal to the part processing time multiplied by the lot size. The lot service time is the lot processing time plus the lot setup time. The arrival of lots of different part types is typically stochastic so lot flowtime behavior can be modeled using queuing relationships. If a lot of parts arrives and the machine is busy, the lot will have to wait in queue. It is normal to assume that lots in the queue will be processed first-come, first-served (FCFS). If the lot sizes are too small, there will be many setups incurred and the utilization, defined as the proportion of time the machine is busy being set up or processing parts, will be high. The result is that the average number of lots waiting for processing may be high. This means the average lot flowtime, defined to be the lot queue time plus lot service time, will also be very high. This drives up total manufacturing times and inventory costs. If the lot sizes are too large, the machine utilization will be lower but the machine will be committed to producing one part type for a long Figure 1 Single Stage Production with Batch Arrivals Q 1 =??? Q 2 =??? c a Lot queue time Lot flow time Lot service time Processing station Machine Q i = Lot size for part type i c a = Lot interarrival time coefficient of variation c s = Lot service time coefficient of variation c d = Lot interdeparture time coefficient of variation period of time. This will again result in more work in queue and an increase in flowtimes. Lot flowtime behavior is convex with respect to lot sizes. A good objective is to minimize the weighted mean lot flowtimes, W, across all part types by selecting the best lot sizes, Q i, for each part type i. Since all part types have unique characteristics, the best lot size combinations are affected by their relative production characteristics as well as the variability of lot interarrivals. Therefore, the problem can be viewed as one of lot size optimization to be solved using response surface methods. The analytical relationships used to describe this problem are approximate and are not well known. Therefore, this is the type of problem that might well lend itself to experimentation. The model embedded in this spreadsheet tool assumes a lot size selection scenario where a single machine is being used to produce batches of two part types. The configuration of interest is illustrated by Figure 1. The lot flowtime relationships embedded in the spreadsheet model are given in the appendix. These relationships may be of interest to operations management or industrial engineering students. However, it is not essential to know anything about the underlying relationships in order to use this tool if the primary interest is to learn DOE methodology. In other words, experimentation can be done in a context-free manner. 3. The Spreadsheet Model The spreadsheet implementation is designed to be simple, transparent, and easily modified. An Excel workbook 1 serves as the user interface for specifying inputs as well as for extracting the experimental results. The user inputs are specified in the Inputs worksheet shown as Figure 2. The colored cells are user-defined inputs. 1 An example of such a workbook (DOE_Tools.xls) can be found at http://ite.pubs.informs.org/. c s c d

INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS 57 Figure 2 Example of Inputs Worksheet Cells in Range D10:E13 specify the production scenario. The user must specify the demand per unit time for each of the part types, D, the time required to set up the machine for each specific part type,, the production rate per unit time, P, and the production lot size, Q. Note that P is the production rate when the machine is steadily processing the specified part type. Since there will be some idle time as well as time for setups, the value of P must be larger than D in order to have a stable system in which all demand is met. The final user input describing the production scenario is in Cell D6. This value specifies the amount of variability in the stream of lot arrivals to the machine. It is expressed as a coefficient of variation, defined to be the standard deviation of interarrival times divided by the mean interarrival time. A higher value indicates greater variability in interarrivals and a longer average queue time. Values in the range of 0.10 to 1.0 are typically appropriate, with 0.30 often being a good guess for practical purposes. Formulas in Range D16:D20, which are based on the relationships found in the appendix, reference the user input values described. Of particular interest is the estimated value of machine utilization. This value must be less than 1.0 in order to obtain feasible performance. When a structured set of experiments is run, as described next, new lot size values will be written into Cells D13 and E13 automatically, based on the user-defined design. In other words, Cells D13 and E13 can be used as inputs to estimate behavior while designing the experiment, but the values in these cells are over-written when the experiment is run. The cells in Range I6:J7 are used to specify the low and high factor settings for two-factor, two-level (2 2 experimental designs. These are referred to as factorial design points. To run a single factor experiment, one would set the low ( 1) and high (+1) settings equal for one of the lot sizes. The cells in Range I11:I13 are used to specify the experimental design. These inputs represent the num- Figure 3CCD with Coded Variables 1.41, 0 Axial points 1, +1 Axial points 0, +1.41 Center points Q 2 +1, +1 1, 1 +1, 1 0, 1.41 0, 0 +1.41, 0 Q 1

58 INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS Figure 4 CCD with Actual Variables and Replications 200, 300 (2) 275, 331 (2) 200, 150 (2) 350, 150 (2) 275, 119 (2) Q 2 169, 225 (2) 275, 225 (10) 350, 300 (2) 381, 225 (2) ber of replications to run at the factorial, center and axial points of the design, assuming a Central Composite Design (CCD). Figure 3 shows the coded variable values for a CCD. Values of zero can be entered for the number of center or axial points if a full response surface model is not required. Figure 4 shows the actual design points that would be generated for the input scenario shown in Figure 2. The numbers in brackets identify the number of replications at each design point. Cell D5 specifies the degree of variability that will be observed in the average lot flowtimes, W. As described in the appendix, it is the coefficient of vari- Figure 5 Example of Outputs After Sorting Q 1 ation of the observed lot queue times, W q, when multiple replications are run. Therefore, increasing this value makes the observed outputs more variable. The Run Experiments button on the Inputs Worksheet runs the experimental design and generates the output. This is done by activating Visual Basic for Application (VBA) macros located within the Excel workbook. In order to make the macros available when loading the workbook, the security level setting should be set to Medium. Changing this setting can be done by going into Tools on the Main Menu bar, then into Macro and finally into Security. As well, running the macros requires that the Analysis ToolPak VBA add-in be available. This can be selected by going into Tools and then Add-Ins. The results for the experimental design will be generated in the Outputs worksheet. The order in which the results appear will be randomized. In other words, the results are presented as if the experimental runs were made randomly. Randomization is good practice in reality since it helps ensure independence by reducing the chance of systematic errors. The Sort button in the Outputs worksheet can then be used to rearrange the output into a more structured form. The sorted data can be readily transferred to other software packages for analysis.figure 5 shows an example of the sorted outputs generated using the experiment specified in Figure 2.

INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS 59 Figure 6 Macro Code Within the VB Editor The macros can be accessed by going into the Visual Basic Editor, located under Tools and then Macro. Figure 6 shows a view from within the VB Editor. In this figure the Project Explorer is visible and the contents of the workbook is shown in the upper lefthand window. If this window is not visible, it can be accessed within the VB Editor by going into View and then activating the Project Explorer. The Project Explorer window should include the atpvbaen.xls file and funcres references. If these are not present, they can be added within the VB Editor to a list found under Tools and then References. This list should also include VBA and the Microsoft Excel Object Library as references. All of the VBA code in the workbook is contained within a module called Experimental Design. This module contains three macros, called ExpDesign, Experiment, and Sort. Part of the code in these macros is shown in the large window of Figure 6. It is not necessary to understand this code unless the user wishes to modify it. However, a brief description is given as follows. The Run Experiment button activates the ExpDesign macro. This macro systematically chooses points in the experimental design and writes the coded values for the design points into Columns A and B of the Outputs worksheet. As well, appropriate lot size values based on the experimental design point are written into Columns C and D of the Outputs worksheet. For each design point processed, the Experiment macro is also called. This macro writes the values for the current design point into Cells D13 and E13 of the Inputs worksheet. It then calculates the machine utilization rate, average lot service time, average lot queue time, and average lot service time for the given observation using the formulas in Range D16:D20 and writes these values sequentially into rows in the Outputs worksheet. This macro also uses the coefficient of variation in Cell D5 of the Inputs worksheet to specify a multiplier for adjusting the calculated queue time to come up with an observed queue time. A normal distribution is used in generating this multiplier. Once data has been generated for each of the design points, the experimental output is randomized. This occurs at the end of the ExpDesign macro. The Sort button, which activates the Sort macro, in the Outputs sheet can then be used to put the data back in a structured form. In order to analyze the experimental results generated, the appropriate columns from the Outputs worksheet can be copied into statistical analysis software, such as Minitab or Design-Expert (Montgomery 2001). In some cases the user of

60 INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS these statistical packages must specify the desired experimental design and then the software will automatically generate worksheet columns showing appropriate factor settings. This means the user must first generate the experimental design before the responses, W, found in column H can be copied and pasted into the analysis worksheet. 4. An Example Problem Suppose the problem is to determine the lot sizes that will minimize weighted mean lot flowtimes given the demand rate, setup time and production rate information shown in Figure 2. Initially the region in which the optimal lot size combination lies would not be clear. In this case we might start by running a small factorial design, using perhaps two replications for each design point. For example, we might choose to use actual lot sizes of (400 600) and (300 500) for Q 1 and Q 2, respectively. These values would represent the ( 1 +1) coded variable settings. Using two replications at each design point would yield eight observations in total. Analysis of the results obtained (not shown) could be done using ANOVA. Such analysis would probably indicate that the main effects for both lot size factors are significant but that the interaction term Figure 7 Minitab ANOVA for a 2 2 Design is not. Furthermore, it could be easily observed that the experimental region selected is unlikely to contain the lot size combination yielding minimum flowtimes and that the ranges should be moved so both lot sizes are reduced. Interaction or surface plots could be used to verify this. Another set of experiments would then be run. The steepest-descent algorithm could be used in determining what lot size combinations should be examined next. Analysis and experimentation would occur iteratively until it appears in the region within which experiments are being conducted is convex and may contain a minimum. Figure 7 shows analysis, using Minitab, obtained with results from a 2 2 experimental design where the coded ( 1 +1) lot sizes represent actual lot size ranges of (200 350) and (150 300) for Q 1 and Q 2, respectively. The top window in this figure shows the experimental design and responses, copied from the Outputs worksheet. The bottom window shows the ANOVA results. The interaction term is shown to be statistically significant, indicating there may be curvature in the response surface. Figure 8 shows the interaction plot associated with these results. At this point we might decide to add center points to the design and determine if curvature is statistically significant. For example, 10 center points

INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS 61 Figure 8 Mean 0.24 0.23 0.22 0.21 0.20 0.19 Minitab Interaction Plot Interaction plot (data means) for W 150 300 Q 2 Q 1 200 150 could be specified by entering this value in Cell I13 of the Inputs worksheet. If using Minitab for analysis, the experimental design must be initially set up before the results can be pasted from the Excel worksheet to the appropriate Minitab worksheet columns. A Minitab menu path of Stat>DOE>Factorial can be used to create the design, including specification of center points. If the number or order of the design points in Minitab differs from those in the Outputs worksheet, the design in Minitab can be edited as appropriate. If the experiment shown in Figure 7 is Figure 9 Quadratic Model of Response Surface rerun with 10 additional center points, the curvature will be shown to be statistically significant (not shown). A final step would be to add axial points. Figure 5 shows the Outputs worksheet obtained when rerunning the experiment as a CCD with factorial and axial points replicated twice and with 10 center point replications. Figure 9 shows the Minitab analysis when a full quadratic model has been specified. In this case the design was initially set up using the menu path of Stat>DOE>Response Surface. Results show a high R 2 value, statistically significant linear and interaction terms at the 95% confidence level and a lack-offit value that is not significant at the 95% confidence level. To confirm the model fits well, the residuals should also be examined. Figure 10 shows residual plots for this example. Contour and surface plots can be used to confirm whether or not a minimum actually exists within the region being modeled. Figures 11 and 12 show plots for this example, obtained using Minitab. In this case it is obvious the minimum is within the region. The optimal lot size combination appears to be somewhere around 315 and 260 for Q 1 and Q 2, respectively. This could be confirmed using the Response Optimizer in Minitab or similar capability in another package. If the region does not contain the minimum,

62 INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS Figure 10 Percent Frequency Residual Plots 99 90 50 10 Normal probability plot of the residuals 1 0.030 0.015 4.8 3.6 2.4 1.2 0.0 0.02 0.01 0.00 0.01 0.02 0.000 0.015 0.030 0.180 0.195 0.210 0.225 0.240 Residual Fitted value Histogram of the residuals Residual plots for W Residual Residual 0.02 0.01 0.00 0.01 0.02 Residuals versus the fitted values Residuals versus the order of the data 0.02 0.01 0.00 0.01 0.02 2 4 6 8 10 12 14 16 18 20 22 24 26 Residual Observation order it would be necessary to shift the design in the appropriate direction and make further attempts to fit a model that will identify a minimum. 5. Discussion and Conclusions This spreadsheet-based tool has been used effectively in laboratory and homework exercises in DOE elective courses for engineering undergraduate and graduate students. Students are given a handout describing the software, problem and relationships. The information provided is similar to that given in this paper. They are then asked to provide a written report for a given production scenario, showing their path of analysis, final results and conclusions. As well, they are asked to explore the effects of setup time reduction if setup times are cut in half. This requires finding new optimal lot size combinations and determining the impact on performance. The exercise has been given to students with and without providing an instructional computer laboratory. In general, a hands-on computer lab in which students are guided through the solution path for an example problem is valuable. It not only facilitates faster and better understanding of the tool but also improves understanding of the requirements when doing the assigned analysis for evaluation. Issues encountered have usually related to picking lot size ranges that are too small or too large. Encouraging students to think about the problem and evaluate behavior at the bounds of the design space helps alleviate the selection of inappropriate ranges. As well, students sometimes have difficulty initially accepting that quite different lot size combinations are selected by different individuals or groups attempting to find the optimal. This happens because the response surface may be very flat around the optimum. It is valuable for them to observe that while the lot size combinations may be different, the predicted lot flowtimes are nearly equal. In summary, more training of students with appropriate skills in design of experiments and response surface methodologies is clearly required. Figure 11 Q 2 300 250 200 150 Contour Plot Showing Optimal Contour plot of W vs Q 2, Q 1 200 240 280 320 360 Q 1 W < 0.20 0.20 0.22 0.22 0.24 0.24 0.26 0.26 0.28 > 0.28

INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS 63 Figure 12 W 0.275 0.250 0.225 0.200 150 Surface Plot 200 250 Q 2 Surface plot of W vs. Q 2, Q 1 300 350 200 250 300 Q 1 The spreadsheet-based tool presented in this paper introduces a unique approach to support such training. An obvious extension would be to allow simulated experimentation in other contexts besides production lot sizing. Appendix. Modeling Relationships A good objective for lot size selection is to minimize the weighted mean lot flowtimes across all part types using a single resource. Since all part types have unique production characteristics, the best lot size combinations are affected by the relative characteristics as well as the variability of lot interarrivals. Furthermore, since there is no closed form solution for lot flowtimes under GI/G/1 queuing assumptions, we must resort to approximations. These approximations are found in the rapid modeling literature, where queuing heuristics are used to approximate work flow in complex manufacturing networks. The following literature includes relevant discussions: Whitt (1983), Buzacott and Shanthikumar (1993), and Hopp and Spearman (2001). The mean lot flowtime for part type i, W i, expressed as the sum of the expected queue time, W q, and the mean lot service time, x i, can be approximated as follows: W i = W q + x i = x c2 a + c2 s 2 1 + x i where x is the weighted mean lot service time for all part types, x i is the mean lot service time for part type i, and is the resource utilization rate. The c a and c s variables are the coefficient of variation of the lot interarrival times and lot service times, respectively. Note that it is common to assume that the average queue time for lots of all types of parts will be equal when using the rapid modeling approach. Therefore, W q is not indexed by part type. The mean lot service time across m part types, including lot setup and processing times, is given by the following: m i=1 x = D i/q i i + Q i /P i m i=1 D i/q i where D i is the demand rate, P i is the production rate, Q i is the lot size, and i is the lot setup time for part type i. Similarly, the utilization rate,, is determined by the following: m = i=1 [ Di Q i ( i + Q )] m i = P i i=1 [ Di P i + i D i Q i ] 0 <1 Note that in analysis using rapid modeling relationships, is usually constrained to be 0.95 or less. Although the utilization must simply be less than 1.00 in order to attain feasibility, it has been found that performance estimates deteriorate very quickly as utilization levels approach 100%. If it is assumed that there is no variation in the lot setup times for a given part type, the standard deviation of lot service times, s, will be a function of lot processing times only. The value of cs 2 can therefore be determined from the following: c 2 s = s 2 x = E x2 x 2 2 x 2 If it is further assumed that there is no variation in processing times for a given part type, the previous equation can be restated as follows: c 2 s = { m i=1 D i/q i i + Q i /P i 2 m i=1 D i/q i x 2 } 1 Using the previous equations, the mean lot flow time for a given part type, i, can be derived as follows: W i = W q + x i = x2 m i=1 D i/q i ca 2 1 + m i=1 D i/q i i + Q i /P i 2 2 1 ( + i + Q ) i P i The weighted mean lot flow time across all part types is then given by the following: W = W q + x The mean weighted lot flowtimes are convex with respect to the lot sizes. This makes the model useful for demonstrating various types of models, including response surfaces. However, any given combination of lot sizes will always produce the same flowtime values when these approximations are used. In other words, there is no uncertainty so replication would serve no purpose. Therefore, the approach taken in using these relationships for DOE training purposes is to induce uncertainty. Uncertainty is induced by adding a random value to the queue time, W q. The value of W q is first calculated for the center point of the experimental design, W q cp. This value is then multiplied by a random variable drawn from a normal distribution [N 1 CV ], with a mean of 1 and standard deviation of CV, where CV is specified by the user in Cell D5 of the Inputs worksheet. The difference between W q cp N 1 CV and W q cp is then the amount of induced uncertainty in average observed lot flowtimes from replication to replication. Therefore, the average observed lot flowtime, W, obtained for any given replication is determined by the following: W = W q + x = W q + W q cp N 1 CV 1 + x

64 INFORMS Transactions on Education 8(2), pp. 55 64, 2008 INFORMS Since the amount of expected uncertainty is always based on the center point of the design, it will be the same at all factor settings within a given design. This helps ensure the ANOVA assumption of equal variances at all combinations of factor settings will not be violated. References Box, G. E. P. 1999. Statistics as a catalyst to learning by scientific method part II A discussion. J. Quality Tech. 31(1) 16 29. Box, G. E. P., P. Y. T. Liu. 1999. Statistics as a catalyst to learning by scientific method part I An example. J. Quality Tech. 31(1) 1 15. Buzacott, J. A., J. G. Shanthikumar. 1993. Stochastic Models of Manufacturing Systems. Prentice-Hall, Englewood Cliffs, NJ. Hopp, W. J., M. L. Spearman. 2001. Factory Physics. McGraw-Hill, Boston. Montgomery, D. C. 2001. Design and Analysis of Experiments, 5th ed. John Wiley and Sons, New York. Whitt, W. 1983. The queueing network analyzer. Bell Systems Tech. J. 62(9) 2779 2813.