Designing a First Experiment: A Project for Design of Experiment Courses

Designing a First Experiment: A Project for Design of Experiment Courses C. M. ANDERSON-COOK A project suitable for use as a first and last assignment given in an introductory experimental design course is outlined, and its implementation discussed. Phase 1 of the project is designed to be given at the beginning of a first course in experimental design and involves designing an experiment within instructor-specified parameters based initially on intuition and common sense. Phase 2, at the conclusion of the course, represents the final assignment of the course and encourages the student to critique their first design, suggest a new and improved design based on their newly acquired knowledge, and finally conduct an analysis of computer generated data (here using MINITAB, although choice of package is flexible) from an actual run of their experiment. KEY WORDS: Blocking; Randomization; Replication; Undergraduate teaching. 1. INTRODUCTION Sir Ronald Fisher advocated that a statistician ought to strive above all to acquire versatility and resourcefulness, based on a repertoire of tried procedures (Box 1978). By encouraging students in an upper level honors class to use their newly acquired skills in experimental design through a project, a first attempt is made at attaining the goal presented by Fisher. Conveying the concepts of design and analysis of experiments can appear to be straightforward, and yet getting students prepared to actually design their first experiment requires a different set of skills than is frequently emphasized in introductory experimental design courses. Indeed, now industrial short courses in the subject matter frequently involve some hands-on experimentation with statapult or some other device. Important ideas such as choice of factor settings, randomization, replication, and interactions can be brought to life and illustrated powerfully to students as they compare an experiment designed using common sense to one based on statistical principles. The selection of the appropriate [design and] method of analysis for a given problem is perhaps the hardest thing to learn and does not usually follow from technique-oriented C.M. Anderson-Cook is Assistant Professor, Department of Statistics, Virginia Tech, Blacksburg, VA 24061-0439 (E-mail: candcook@vt.edu). The author thanks David Bellhouse, Department of Statistics, University of Western Ontario, London, Ontario, Canada, and a reviewer for their careful comments, which substantially improved the presentation of the article. courses (Chatfield 1982) but is encouraged through the use of this project. Students entering into a one-semester Design of Experiments course might typically have taken one full course in introductory statistics and are majoring or minoring in statistics. The idea for this project originated when I was planning an outline for this course while working at the University of Western Ontario, London, Ontario, Canada. I realized that the standard approaches to the subject matter frequently do not actively involve the learner in some important aspects of the design process. Texts such as Box, Hunter, and Hunter (1978), Montgomery (1991), and even some introductory texts such as Wardrop (1995) discuss the importance of decision making and implementation of the physical details of planning and executing a successful experiment and encourage instructors to actively incorporate this aspect into experimental design courses. The critical message that students must internalize is that no amount of statistical wizardry can salvage an experiment where data quality is poor, or that graphical or qualitative methods may be the only analyses possible from a poorly designed experiment. Another motivation for the project is that some of the key concepts may not seem as important when discussed in abstract terms. Hunter (1976) gave a comprehensive list of simple but highly creative experiments that students have performed and analyzed. Appendix 11A in Box, Hunter, and Hunter (1978) gives details of a factorial experiment that could be conducted as a student project. This article suggests an alternative to a complete experiment, with a two-part project to design, critique, and re-design a common experiment. The project strives to accomplish the following objectives: 1. Illustrate the standard framework in which many experiments are designed a fixed set of resources to be used to gain the best answers possible to some questions about the relationship between input factors and response. 2. Allow students to show their creativity and intuition in designing the initial experiment. 3. Force students to think about the logistical details of running an experiment and understand the inherent complications of collecting high-quality data. 4. Demonstrate need for randomization and replication so that an experiment can be analyzed using the standard factor main and interaction effects model. 5. Allow students to critique a typical poorly designed experiment to assess its strengths and weaknesses. 6. Provide an open-ended question for students to select an appropriate experimental design from a collection of studied methods. 338 The American Statistician, November 1998, Vol. 52, No. 4 c 1998 American Statistical Association

7. Analyze an experiment from start to finish and discuss the conclusions in largely nonstatistical terms for a potential client. 8. Provide work in undergraduate education as a means of combating the tendency of students to compartmentalize their studies with little or no synthesis between disciplines (discussed in Anderson and Loynes 1987). 2. DETAILS OF PROJECT The project encompasses many aspects throughout the course the actual assignments, the office hours when students are encouraged to drop by to discuss their proposals, and in-class discussion during and after the assignments. The following is the set of instructions given to the students as Assignment #1 at the start of the course. It is usually due within three weeks of the start of classes, and at which time general discussions of replication, randomization, and blocking have already taken place during lecture time. Assignment #1 This is an exercise in planning an experiment. Think of yourself as a consultant hired by a scientist trying to answer a scientific problem. Because of the cost and time required to actually conduct the experiment, we will not be actually collecting the data, but will consider all aspects of design short of that. You may work in groups of two or alone and should hand in two (2) copies of your report. The purpose of the experiment is to study what influences the rate at which an ice cube is cut in half by a piece of wire as it melts. The results of the experiment will be used to predict how long a future ice cube will take to be cut in half under a given set of conditions. The diagram (Fig. 1) shows how each ice cube would be tested for a given set of conditions. The ice cubes are prepared, put on the rack, and a time is recorded from the start time until the weight drops and lands in the metal pan below. Because of the size of the rack, a maximum of 12 ice cubes can be tested at the same time. A number of factors are thought to potentially influence the rate of melting. The three factors to be considered in this experiment are the influence of salt, the diameter of the wire used, and the size of the weight attached to the wire. Your assignment is to design an experiment that you feel will allow you to draw some conclusions about each of these factors. Your budget for the experiment is $1,200 with a good supply of weights, wires, and salt already having been purchased and available for use. The main cost of the exper- Figure 1. Diagram of Ice Cube Tray Layout iment is in labor (this is quite common) since the technician who has been hired to run the experiment charges $20 per ice cube observed (i.e., only 60 ice cubes can be tested total). The experiment will be run during working hours (9 a.m. to 5 p.m.) over seven consecutive days (Monday to Sunday). You can assume that no single ice cube will take more than two hours to melt. There are 10 sizes of weights available (.2kg,.4kg,..., 2kg) and 10 sizes of wire (.3mm,.4mm,..., 1.2mm). It is also felt that values between 0 and 1 gram of salt per ice cube are usefully used for enhancing melting (You may choose any value in that range that you wish). It would be good if at the end of the experiment, some general conclusions could be drawn about the influence of wire width, weights, and salt in the ranges described. Things to include in the discussion (your report should be no more than five pages in length): 1. The total number of runs do not feel obliged to use the whole budget if you feel you can get enough information from a smaller number of ice cubes. 2. The size of wire, weight, and amount of salt to be used for each of the ice cubes tested. (A summary table may be helpful for this.) 3. Special precautions for the technician to take while running the experiment (i.e., things to observe or monitor, or other instructions). 4. Potential problems that you anticipate in running the experiment. 5. Some idea of how well you feel you have fulfilled the mandate of the experiment. First, a few comments about some of the particulars of the assignment. A model of the melting tray with pan is brought into class to help the students visualize the experimental conditions. While the students are receiving instructions about the details of the project an ice cube is set-up and monitored. Although not essential, the inclusion of at least one actual observation provides students with a better understanding for some of the inherent problems that might be encountered in the actual experiment. Working in pairs encourages considerably more discussion of potential issues and design possibilities, with the fringe benefit of reduced evaluation time. The assignments handed in by a group are generally more than twice as complete as those done alone, demonstrating the benefit of discussion and teamwork. Ledolter (1995) talked of the benefits of team learning, and I feel very strongly that encouraging verbalization of statistical concepts is an important aspect of learning. In addition, current pedagogy advocates a variety of learning opportunities in classes, and a group assignment that is discussion based is a good complement to the largely mathematical content of many statistical courses. One of the two copies handed in is returned, while the other is kept by the marker for use when evaluating the final assignment. Students initially require strong assurance that the marking of the first assignment will not be based on the right answer, but rather on how well they have thought through the details of running the experiment and their explanation of the thought processes involved in de- The American Statistician, November 1998, Vol. 52, No. 4 339

ciding how the design was chosen. These explanations are a useful reminder of the motivation for the layout, when in the final phase of the project the students are asked to evaluate the design used. When the assignments are collected, each pair of students is asked to fill out a three-dimensional grid, where each axis corresponds to one of the factor effects (Fig. 2), marking the combination of factor levels where their experimental observations are to be taken. For many students it is quite revealing to think about the choices of factor settings as points in a three-dimensional space. Many students express surprise at the coverage that their design provides, when viewed in this manner. This exercise, while helpful for the students, also provides a quick overview of the strategy used to select factor combinations for the marker. The evaluation of Assignment #1 is quite subjective, with the major emphasis on completeness, discussion of instructions to the technician (could a worker actually follow the instructions outlined?), and consideration of potential problems, as these are an indicator of the investment of time and thought made by the students to consider the experimental issues. Some of the main issues that should be brought out in the discussion by students are: choice of factor combinations (want evidence that the range of all factors was considered); standardization (such as controlling the size of ice cubes, temperature of the lab, handling procedures for ice cubes, etc.); timing of experimental runs (logistics of starting multiple ice cubes simultaneously); monitoring of noncontrollable factors (measuring temperature, humidity, time of day, proximity of ice cubes to each other, etc.) to check on their possible influence; limitations of the scope of their results (conclusions restricted to comparison of relative effect of factors, or specific influence of factors under specific lab conditions); and Figure 2. Three-Dimensional Grid of Factor Levels. variability of results under same test conditions (inclusion of at least one repeated run to monitor within combination variation). Since this is their first attempt at designing a complete experiment and the purpose of the first phase is to lay the groundwork to motivate the course by illustrating the need for formal design strategies, some latitude is given for those assignments that overlooked some of these issues. Overall, this assignment is primarily a motivational tool for subsequent topics considered in the course. When this phase of the project is returned to the students, the class engages in a lengthy discussion of issues associated with giving instructions for carrying out an experiment, potential problems if instructions are vague or incomplete and how the notions of blocking, randomization, and replication are integral to the success of an experiment. During the term, the standard designs, such as one-way, two-way, randomized complete block, factorial, fractional factorial, and Latin and Graeco-Latin square designs are typically discussed. In addition, fixed and random effects models are considered. As a result, the students have quite a large selection of methods available to consider when the final phase of the project is assigned. Statement of the final assignment appears in the following. Students typicially have several weeks to complete the assignment. Assignment #(Last) Part 1: (25%) Write a one (1) page analysis of your first experiment. Comment on how well you feel that the experiment would be able to answer the mandate for which it was designed. Make sure you incorporate information that you have learned about the important concepts of experimental design. At this stage of examining the problem, focus on the actual design, not how well you addressed the actual implementation of the experiment. (In other words, you should be commenting on your initial choice of settings for weights, wire, and salt, not on how the experimenter should handle the ice cubes or measure the time taken until they drop). Part 2: (75%) Now tackle the problem again using the understanding that you have gained from the course. Design a new experiment to answer the same mandate as the original problem (i.e., we want to find out what relationship there is between weights, wire, and salt on the time taken to melt an ice cube). You can omit the explanation about how the experimenter should implement the running of the experiment, but you should give a table for the order, time, and setting of the (up to) 60 runs. Also give a brief paragraph about how you choose the set-up and the details of how you ended up the particular order in which the ice cubes will be tested in the final table. (This part should be no more than a page + table.) Visit the teaching assistant during one of the appointed times with a MINITAB spreadsheet of your design (one column for each of Run Number, Salt, Wire, Weight ) to obtain data for your experiment. Do a complete analy- 340 Teacher s Corner

Table 1. Sample of Summary Table Layout Run # Time of run Amount of salt Wire width Weight size 1 Mon. 10 a.m. 1 g.3 mm 2 kg sis of the experiment, and comment on the conclusions you would draw about the three factors and their effect on ice cube time melting, and the model that your choice of experiment implies. These comments and conclusions should be understandable for the scientist who hired you. Write up a report incorporating any graphs, tables, MINITAB output you feel are useful. (No reasonable limit on size of this part but do not include things that are not important to your analysis.) Again a few comments about the particulars of this phase: Many students are initially too ambitious in their attempts to design overly complex experiments, and need to be reminded that they should design an experiment for which they know how to analyze the results. In previous versions of the instructions, the sentence instructing the students to itemize the run-order of the observations was more vague and only hinted at randomization. Since a more explicit statement about run-order was included, a greater proportion of the class correctly incorporates this facet into their design. The actual generation of the data can easily be done by the teaching assistant in approximately five minutes per group, by using a pre-specified equation to multiply the factor combination levels by known coefficients (I usually include at least one interaction term and a small timedependent term to influence their residual analysis), before adding a random error term. Therefore, the data generating the data is of the same form as the assumed underlying model (except for the time-dependent term), and the magnitude of the error term is chosen relative to the coefficients of effect terms. Again when collecting the assignments, a copy of Figure 2 is filled out to give a quick overview of the design used. Evaluation of this phase contributes a significant portion to the student s overall grade, since it provides an opportunity to incorporate many of the theoretical and practical aspects discussed throughout the course. Typically, the students find that the critique of the first experiment is quite difficult. The issues of randomization, replication, and blocking should all be discussed at this time, and comments made about the experiment s ability to test interactions, and what sort of analysis is possible given data for the proposed set-up. The majority of students have designed an experiment in Phase 1 that they are not able to analyze, since it is not randomized or replicated sufficiently. In the design portion, the emphases should be on the incorporation of randomization, replication, choice of design, and designation of factor effects as fixed or random. The students provide a statement of initial and final models from this experiment, analysis of variance table, expect mean squares for all ANOVA terms, and residual analysis in the analysis portion of the assignment. The discussion and conclusions portion is very important since it has been emphasized in the course as the connection between statistician and scientist to convey meaningful information about the results of the experiment. 3. ASSESSMENT OF THE PROJECT S EFFECTIVENESS As a teaching tool, I find that the project outlined gives the students an opportunity to combine their statistical knowledge with common sense to work on a problem that is not contrived in its simplicity, but not overwhelming in complexity. While many of the aspects and goals of the experiment are similar to those described by Hunter (1976), there are some major differences that I see between this approach and that of actually conducting the experiment: Phase 1 provides both a motivational portion at the beginning of the course to encourage students to think about the concepts in a practical way, as well as allowing them later to critique a typical intuitive design. It also provides a good frame of reference throughout the course, when discussing new designs or analyses. Describing the implementation of the experiment, instead of doing the collection themselves, emphasizes the skill of being able to give clear, detailed instructions. While ideally it would be helpful to have the students collect the data as well, the exercise of describing the experiment is also highly beneficial. If time and resources were available to allow the students to collect their own data, I think this could be a useful further supplement to the exercise. Having all of the students work on the same basic problem and come up with several correct answers encourages discussion of the relative merits of designs and common pitfalls that might be encountered. By simulating the data, there is a bit more control for the instructor over what type of effects should be detected and incorporated into the final model. At the conclusion of the course, students do not have the tools to adequately deal with missing data, and so this possible complication is avoided with simulated data. (For a second course in experimental design, actual collection of data would allow students the chance to do an analysis with some additional complications.) By including the time trend in the data, students can see that even though all the same effects are present in all of their experiments, results vary and the devastating effects of not randomizing are emphasized. Phase 2 of the project gives some closure to the course by incorporating many of the key concepts. It also serves as a useful tool to illustrate any confusion with these ideas before the final exam. As a result, the assignments are returned to the class a week prior to the final exam. This allows for a final discussion on some of the key aspects of the experiments and what the students have learned, and allows students to correct any major misconceptions before the final evaluation of the course. The project may be the The American Statistician, November 1998, Vol. 52, No. 4 341

first opportunity for students to demonstrate their statistical competence in a context that is regarded by the class as a realistic problem that might be encountered in the course of their future studies or work experience. There is sufficient flexibility in the constraints of the experiment that a variety of sound designs are suggested. I have recently started using phase 1 of the project in a graduate experimental design course as a refresher for the students to review their background knowledge in this area, and to help them confront some of the practical issues that they might face in a consulting role if asked to design a simple experiment. The additional time commitment required by an instructor to implement both phases of the project would be approximately equivalent to marking half of a class set of two-hour midterms, and as such is a valuable exercise for the students without unreasonable time investment by the instructor. The overall reaction of the students is positive, since the process and the evaluation provide them with some satisfaction. The assessment is perceived to reward both creative open-ended thinking and application of learned methods, and the process encourages the students to use all the major ideas from the course in a single project to produce a final product that is perceived as valuable and marketable. Comments from the students include sentiments that they now feel more confident about their understanding of statistics and its implementation, and that the subject matter is interesting and worthwhile. Overall, the quality of the projects is very high and demonstrates good understanding of the important issues of design and analysis of experiments. [Received January 1996. Revised April 1998.] REFERENCES Anderson, C.W., and Loynes, R.M. (1987), The Teaching of Practical Statistics, New York: Wiley. Box, G.E.P. (1978), R.A. Fisher: The Life of a Scientist, New York: Wiley. Box, G.E.P., Hunter, W.G., and Hunter, J.S. (1978), Statistics for Experimenters, New York: Wiley. Chatfield, C. (1982), Teaching a Course in Applied Statistics, Applied Statistics, 31, 272 289. Hunter, W.G. (1976), Some Ideas about Teaching Design of Experiments, with 25 Examples of Experiments Conducted by Students, The American Statistician, 31, 12 17. Ledolter, J. (1995), Projects in Introductory Statistics Courses, The American Statistician, 49, 364 367. Montgomery, D.C. (1991), Design and Analysis of Experiments (3rd ed), New York: Wiley. Wardrop, R.L. (1995), Statistics: Learning in the Presence of Variation, New York: Brown Publishers. 342 Teacher s Corner