Design of Experiments To help us work through the DOE steps, we re going to assume that we work for a company that produces bicycle tires where we re particularly interested in learning how to maximize the puncture resistance of our tires. Step 1: Identify Factors/Levels and Choose Appropriate DOE Design The first step to the DOE process is to identify the factors we ll study, as well as, the levels of these factors. Once this is done, we ll then select the appropriate DOE design. To be sure, this is an extremely important step of the process. As such, we d like to offer a few suggestions for how to go about selecting the factors and choosing their levels. First as we ve mentioned earlier, we can and should use tools like process maps, the C&E Matrix, and FMEA to help us choose the critical factors. We can also use Screening DOEs, to help us identify critical factors worthy of Full Factorial Design study. Additionally, Hypothesis Tests which can usually be done quickly and efficiently can be very helpful in identifying factors that seem to have a strong effect. Last, but certainly not least, we should always leverage the many years of process knowledge we have all around us each and every day. Design of Experiments GembaAcademy.com 1
Once we have the factors identified, we need to choose the levels we ll test during the DOE. The key is to ensure that we choose levels big enough to see a difference while still being realistic. For example, we d never want to set a factor to an unsafe level or to a level that could potentially harm equipment. Once again, this is where process knowledge and subject matter expertise really comes in. In our bicycle tire example, the team used the results of their C&E Matrix and FMEA to help them narrow the list of potential factors down. They then discussed these factors with machine operators and supervisors, as well as, several process engineers until everyone agreed that these three factors were the best options - temperature, cure time, and the density of the rubber used. They also discussed and agreed upon the levels for each factor. For example, the low level for temperature is 400 and the high level is 450. Again, the response or output variable being studied is puncture resistance. It should be noted, the team already performed a measurement system analysis on the machine that will be used to measure puncture resistance in order to ensure they can trust the data. Here s what the initial Yates Diagram looks like for this study. The first two columns represent the standard order and run order. Typically, we ll always randomize our designs meaning the Minitab standard order, which is the normal non- randomized order, will be different than the actual run order. For teaching purposes, we disabled randomization which is why they match. Steps to Full Factorial Design of Experiments GembaAcademy.com 2
We then see the three factors and their levels presented in the next three columns. As you can see, run 1 has us setting temperature to 400, and cure time to 30, and density to 10. Once a tire is produced, we ll measure and record how much pressure is required to puncture the tire here. No matter what kind of design we create, Minitab will always use one run to estimate the intercept of the predictive equation much like we saw when working with regression. Unfortunately, when we only have a single replicate of a Full Factorial Design meaning all the combinations are tested one time, all the runs are what we refer to as being spent. Replication In other words, there s no way to estimate statistical error. As such, P values won t be created making it more difficult to determine which factors are statistically significant. In order to resolve this problem, we can replicate the design. A Replicate is a duplicate run of all factor level combinations in a DOE. This allows us to estimate statistical error and calculate P values. It is possible to replicate a design as many times as we d like, but obviously each replicate increases the time and cost of the experiment so this must be taken into consideration when deciding how many replicates to choose. In our bicycle tire example, we chose to replicate the design two times. The first 8 runs represent the first replicate and the last 8 runs, which are identical to the Design of Experiments GembaAcademy.com 3
first 8. Again we ll want to randomize our design when doing this sort of experiment. To show you what this looks like, here s what this design looks like after enabling randomization in Minitab. Notice how the Standard Order and Run Order columns no longer match. The combinations of factors and levels haven t been changed, but the suggested order of working through the experiment has. Blocking A close cousin to Replication is Blocking. Blocking is an experimental technique that groups runs into logical collections in order to account for unavoidable process variation. For example, let s imagine we re running an experiment in a factory that operates on two shifts. In order to ensure the shift doesn t impact the experiment, we could block on shift. In other words, half of the experiment would be done on day shift and the other half would be done on night shift. Minitab would then tell us whether this blocking variable was significant and even if it is, since the design was evenly spread across both shifts, the results would still be valid. In fact, some would say the results would be more robust since the variability across shifts was included in the experiment. Of course if we do see a statistical difference between blocked variables, we d want to understand why this is meaning we d definitely investigate the situation. Step 2: Plan and Prepare for the Experiment Once our design has been created, it s time for step 2, plan and prepare for the experiment. This is another very important, yet sometimes neglected, step in the DOE process. First, it s important to plan and document everything related to the experiment. Design of Experiments GembaAcademy.com 4
We ll want to answer questions like where will the experiment be done? Who will help us run the experiment? Where will we get the raw materials needed and who will pay for everything? We ll also want to communicate with all stakeholders including the people who do the work on a regular basis. Obviously if the experiment will interrupt the normal work flow, this downtime will obviously need to be planned for and approved. Next, we ll want to do our very best to control the environment around us in order to minimize so- called noise variables. Things like the temperature in the room or humidity are examples of variables that can add noise and instability into the experiment. Of course like we mentioned earlier, we can always use Blocking to combat this. The fourth tip is to always randomize the experiment since randomizing the experiment is the best way to combat any potential noise in the experiment. The fifth tip is to always practice the DOE before running the actual DOE. Now I ve personally done hundreds of DOEs and feel like I know what I m doing, but no matter what if I don t do a few practice runs before the actual DOE, I always pay for it. You see running an experiment requires excellent organization and planning, and the only way to prepare for this is to practice. Last, but certainly not least, once the DOE is complete and we feel like we ve learned how each factor behaves, we always want to confirm the results with one last confirmation run. We ll learn a lot more about this once we cover Optimization Designs. Design of Experiments GembaAcademy.com 5
Step 3: Run the Experiment Once we ve fully prepared and practiced, it s time to run the experiment. In other words, it s game day! Hopefully everything goes extremely smooth with every experiment you ever run, but chances are good things will pop up and issues will arise. This is why it s important to have, or quickly create, contingency plans. For example, I once worked with a team that had planned a 16 run DOE. We were just about halfway through and disaster struck. Our measurement system stopped working which definitely didn t happen during our practice runs. After a quick talk, we decided to press on with the experiment and measure the parts later once the measurement system was working and proven to be repeatable and reproducible. Here s what our experimental table looked like once all the puncture resistance data had been added. Step 4: Analyze and Interpret Results At this point we re ready for step 4, analyze and interpret results. To start things off, we need to tell Minitab which terms we want to include in the model. When we start, we always include them all. As you can see, we have the three main effects; Temperature, Cure Time, and Density selected. We also have the three Two- Way Interactions selected which includes AxB which represents the Temp x Cure Time interaction. Lastly, we also have the Three- Way Interaction of Temp x Cure Time x Density selected. Design of Experiments GembaAcademy.com 6
Here s the first part of the Minitab statistical output. We see some R- Squared values which help us understand how well our design did at modeling the variation in the experiment. With an R- Squared Adjusted value of 98.23%, we can feel pretty good that this design did a nice job. R- Squared Predicted R- Squared Predicted is a statistic we haven t seen before. This value simply tells us how well the calculated model predicts the response. Next, we see P- values which help us determine if any of our main effects or interactions are significant. Like ANOVA, the null hypothesis is that there are no significant main effects or interactions and the alternate hypothesis is that there are significant main effects or interactions. As we can see here, the main effects of Temperature and Cure Time are definitely significant with a P- value of 0. We also see the two way interaction of Temp x Density is also significant even though the Density main effect on its own, isn t significant. This is a perfect example of how powerful a DOE can be since there s literally no way we would have ever discovered this interaction with a One Factor at A Time experiment. Design of Experiments GembaAcademy.com 7
All the other P- values are greater than.05, meaning they aren t significant factors or interactions. Minitab also includes a more detailed ANOVA table as shown here where they do a really nice job of breaking down each source. For example, they break the main effects down in this section, where again, we see that Temp and Cure Time are significant with P- values of 0 as is the two- way interaction of Temp x Density. I do want to point out that this analysis has been done in coded units, which is the default setting. In other words, Minitab used - 1 and +1 for the low and high factor levels in the design. Doing this makes the design perfectly balanced enabling us to remove the insignificant factors from the model. They also present the coefficients in uncoded form in this section in case you want to build a predictive model which does require the coefficients to be in their uncoded format. Minitab also gives us a nice Pareto Chart summarizing the effects. The red line represents the P- value at.05. Pareto Chart of the Standardized Effects (response is Puncture Resistance, Alpha = 0.05) 2.31 B A Factor A B C Name Temp Cure Time Density AC Term BC ABC AB C 0 5 10 15 Standardized Effect 20 Design of Experiments GembaAcademy.com 8
Any bar that extends beyond this red line is significant. Again we see that factors B, which is the Cure Time, A, which is Temp, and the interaction of A x C are significant. Reducing the Model Now that we know which main effects and interactions are significant, it s time to reduce the model accordingly. To do this, we simply select each main effect and interaction accordingly as shown here. You might wonder why we ve included the main effect of C, or Density, even though it wasn t found to be significant. As it turns out, since the interaction of A x C is significant, we have to include the main effect of C in the overall model. In fact, Minitab will give you an error if you don t. Here s the Minitab output of the reduced model. We once again see Temp and Cure Time are significant as is the interaction of Temp x Density. Design of Experiments GembaAcademy.com 9
Main Effects and Interaction Plots Next, once we ve reduced the model as far as we can, we ll want to create Main Effects and Interaction Plots. Here s what the Main Effects Plot looks like for this study. As you can see to maximize puncture resistance, we d want to set Temperature to 450 and Cure Time to 30. Main Effects Plot for Puncture Resistance Data Means Temp Cure Time 890 880 870 860 850 Mean 400 Density 450 30 50 890 880 870 860 850 10 12 We also see as a stand alone main effect, it doesn t seem to matter which density level is used, but let s hold that thought for a bit and look at the interaction graphs. Since we have three factors, it can be a little confusing to interpret what we re looking at. Let s break it down to see if we can make sense of it all. In this section, we see Cure Time at levels 30 and 50 compared to Temp at 400 and 450. The red line represents when Temp is set to 450 and the black line represents when Temp is set to 400. This data point was created by averaging the responses every time Temperature was 450 and Cure Time was 30. Design of Experiments GembaAcademy.com 10
This data point was created by averaging the responses every time Temp was 400 and Cure Time is 50. Since the lines are mostly parallel to one another, we know the interaction isn t significant which we of course already knew since we looked at the P- values earlier. Next, this section shows us the interaction of Density by Cure Time. In this scenario, the red line represents when Cure Time was 50 and the black line represents when Cure Time was 30. Again since the lines are mostly parallel, we can tell there's no interaction. Lastly, here we see the interaction of Temp x Density which as we already know from our P- value analysis is significant. Notice how the lines are no longer parallel meaning these factors do have a significant interaction. Since our goal is to maximize puncture resistance, it s clear the red line, which represents a temperature of 450, is the best level. What s also interesting is that the combination of Temp at 450 and Density at 10 does seem to have a slightly better puncture resistance. Assuming there are no other process or cost reasons against it, we d most likely choose a Density of 10. Design of Experiments GembaAcademy.com 11
Finally, before we complete the DOE analysis process, we ll want to examine the residuals just like we ve done during ANOVA and regression analysis. Residual Plots for Puncture Resistance Percent 99 90 50 10 Normal Probability Plot Residual 5.0 2.5 0.0-2.5-5.0 Versus Fits 1-10 -5 0 Residual 5 10 810 840 870 Fitted Value 900 930 Histogram Versus Order 4 5.0 Frequency 3 2 1 Residual 2.5 0.0-2.5-5.0 0-6 -4-2 0 Residual 2 4 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Observation Order We ll first check to see that the residuals are normally distributed, which they seem to be. We ll also want to make sure that they're random which they seem to be in this residuals versus order graph. Lastly, we ll want to check that the variances are even, which again, they seem to be. That s the DOE process. Conclusion Let s summarize what we learned through this particular experiment. We learned that in order to maximize puncture resistance, we should set Temperature to 450, Cure Time to 30, and Density to 10, due to its interaction with Temperature. Design of Experiments GembaAcademy.com 12
Like we mentioned earlier, we d definitely want to run a confirmation trial using these settings, but a quick glance back at the raw data from this experiment shows that this 450, 30, 10 combination definitely returned the highest responses at 924 and 920 respectively. During our confirmation run, we d expect our puncture resistance to be very close to these values. Design of Experiments GembaAcademy.com 13