IMPROVE YOUR BUSINESS WITH DESIGNED EXPERIMENTS Dave Northcutt ASQ Certified Quality Engineer INFORMS Certified Analytics Professional dnorthcutt@acm.org May 11, 2016
GOAL & AGENDA Goal: Enable participants to begin to use the power of Designed Experiments in their own work by sharing the basic methods needed for successful experimentation Agenda 1. What are designed experiments and why do I need them? 2. How do I set up, run, and analyze a designed experiment? 3. Do I have to do all this by hand? 4. How do I sell others on this approach? 5. Where can I go for help? 2
WHAT ARE DESIGNED EXPERIMENTS? Virtually every business delivers an output, a product or a service, using some set of inputs Those inputs (factors)can often take on a virtually unlimited number of values (levels), each of which may or may not have an impact on the output (response) Designed experiments are experimental methods that attempt to create the maximum amount of insight at the lowest cost by systematically controlling factor levels Without such techniques, it is possible to overlook key factor combinations, misinterpret results, and fail to persuade others of the proper course 3
UNFORTUNATELY, NOT ALL FACTORS ARE CONTROLLABLE Uncontrollable factors can be a major source of variability in response measures (Y s) Designed experiments should include techniques to deal with uncontrolled variables (Z s) as well as controllable variables (X s) From: DOE Simplified: Practical Tools for Effective Experimentation by Mark J. Anderson and Patrick J. Whitcomb, 2000, p.3. 4
DESIGNED EXPERIMENTS WERE DEVELOPED BY SIR R. A. FISHER (1890-1962) Fisher pioneered the development and use of designed experiments, primarily in agricultural studies Among his many accomplishments are: Analysis of Variance (ANOVA) P-values F distribution Small sample statistics Numerous theorems with his name attached His landmark DOE paper was titled The Differential Effect of Manures on Potatoes 5
THERE ARE IMPORTANT DIFFERENCES BETWEEN INDUSTRIAL EXPERIMENTS AND AGRICULTURAL EXPERIMENTS Agricultural and Medical experiments often have a multiplicity of experimental units (different fields, different offspring, different patients, etc.). There is often only one experimental unit the factory in a business. Agricultural and Medical experiments often take years or even decades to complete, while industrial experiments often need to be done in days or weeks. Industrial experiments often face a large array of factors to be considered, and simply sorting out the active factors from the inert ones is usually a major accomplishment. The emphasis with industrial experiments is on the practical, not the theoretical. 6
BASICS OF DESIGNED EXPERIMENTS Repetition because we are using variation to identify significant differences, the more variation you can observe, the higher your confidence will be Randomization this is the simplest way to account for the Z s (uncontrollable factors). By randomizing the order of the experiments, uncontrollable factors have a better chance of affecting the outcomes equally Blocking when an uncontrollable factor is known and can be measured, designing the experiment so that each level of that factor forms a block of experiments can greatly reduce complexity and improve insight 7
A SIMPLE, FUN EXPERIMENT Three Players must compete for the last slot on the company bowling team. To determine the winner, you ve decided each bowler will roll 6 games. The results are below. Game Pat Mark Shari 1 160 165 166 2 150 180 158 3 140 170 145 4 167 185 161 5 157 195 151 6 148 175 156 Mean 153.7 178.3 156.2 What design considerations did you have prior to the roll off? What is the hypothesis we are testing? Is it clear who should have the last slot on the team? Is it clear who should be the first alternate? 8
ANALYSIS This is a single-factor analysis of variance (ANOVA); repetition and randomization are the two important factors here On the face of it, Mark is the winner, and Shari is the alternate, but the amount of variation is high in some cases Game Pat Mark Shari 1 160 165 166 2 150 180 158 3 140 170 145 4 167 185 161 5 157 195 151 Overall 6 148 175 156 Mean Mean 153.7 178.3 156.2 162.7 Variance 92.27 116.67 54.97 Var_y 184.3426 Var_Pooled 87.96667 F 12.57358 P_value 0.000621 The F-value is used to determine if the means are equal. It is: n*var(y)/varpooled(x) The resulting F-value is then compared to a tabled value, or a p-value can be computed directly It is common to use p=0.05 as a significance cutoff value (1-in- 20 chance). We see the 3 bowlers are not equal But, this is tedious is there an easier way? 9
LET EXCEL DO THE HARD WORK! Excel can do the calculations for you, but you must turn the features on before you can use them start by searching for Add-ins Select the Add-ins item, then click on Analysis ToolPak in the dialog box. The VBA add-in is handy if you do any programming in Excel. Solver is a very powerful Add-in as well. Neither are needed for designed experiments. 10
EXCEL DATA ANALYSIS Choose ANOVA: single-factor under Data Analysis on the Data tab in the ribbon Fill in the dialog box Input Range is the data you wish to analyze The Labels box is a nice feature We chose p=0.05 (Alpha here) as a significance cutoff ; you can use others I like to keep the Output Range on the same Worksheet, but you can control where it goes Click on OK and stand back! 11
EXCEL ANALYSIS: RESULTS Notice, we got the same results from Excel, with fewer formulas to enter! 12
SO, WHO S REALLY THE WINNER? The F-value simply tells us is that the three bowlers are not the same formally stated, we reject H 0 : µ Pat = µ Mark = µ Shari A simple Excel graph will tell us what is going on! Mark s range is clearly above the other two he is the winner Pat and Shari are not distinguishable from these data If you want to choose a first alternate bowler based on scores, you will need to have Pat and Shari roll more games Note: Always insist on a significant F-test before looking at factor plots! 13
BEFORE AND AFTER EXAMPLE Your company produces a unique metal, Flexium, and the folks in your Chemistry Department say they have a new catalyst that will improve yields. The QA head, N. I. Here, says he ran a quick test each way and the results weren t any better. You ve been asked to arbitrate this decision. Flexium sells for $1,500 per pound on the open market, so higher yields would almost certainly increase profits. How do you decide who s right? What design factors would you include? 14
RESULTS OF A 5-SAMPLE EXPERIMENT Sample As-is With Catalyst 1 45.3 49.2 2 44.2 47.5 3 46.1 48.1 4 43.9 46.9 5 45 50.1 mean 44.90 48.36 variance 0.775 1.668 F-Test Two-Sample for Variances With Catalyst As-is Mean 48.36 44.9 Variance 1.668 0.775 Observations 5 5 df 4 4 F 2.152258065 P(F<=f) one-tail 0.238059599 F Critical one-tail 6.388232909 The average seems higher with the Catalyst, but visual tests can be deceiving This can be tested with a simple t-test, but we need to decide whether the variances are equal first they don t appear to be The F-test is not significant, so the variances are assumed equal We will choose the t-test: Two Sample Assuming Equal Variances from the Data Analysis menu We are testing H 0 : µ Catalyst <= µ As-Is (i.e., the Catalyst is no better), so this is a One-tailed test 15
RESULTS OF A 5-SAMPLE EXPERIMENT t-test: Two-Sample Assuming Equal Variances As-is With Catalyst Mean 44.9 48.36 Variance 0.775 1.668 Observations 5 5 Pooled Variance 1.2215 Hypothesized Mean Difference 0 df 8 t Stat -4.949933535 P(T<=t) one-tail 0.000560489 t Critical one-tail 1.859548038 P(T<=t) two-tail 0.001120978 t Critical two-tail 2.306004135 The p-value is significant, so we reject H 0 and conclude that the Catalyst does have a positive effect on yield So, as long as the cost of the Catalyst is lower than the expected increase in revenue, you should change methods 16
WHAT HAPPENS WHEN WE INCREASE THE COMPLEXITY? As the number of factors and levels increases, the number of unique experimental trials grows significantly 2 factors, 2 levels: 4 trials 2 factors, 3 levels: 9 trials 3 factors, 2 levels: 8 trials 3 factors, 3 levels, 27 trials, etc. We need a systematic approach to insure we cover the permutations Changing only one-factor-at-a-time a commonly held belief will miss a lot of possibilities and likely will lead to the wrong answer Let s look at a simple, 2-factor, 2-level example: Your process dries coatings in a conveyor and you can control 2 factors, the heat and speed of the conveyor Both heat and speed can be high or low independently; you are using High Speed and High Heat today The output variable (Y) of interest here is the number of visible cracks in the coated piece 17
THE CONVEYOR EXPERIMENT What are they design considerations? We have decided to make 3 runs at each combination of levels for a total of 12 trials. If we use a simple one-factor-at-a-time approach, we will test HH, HL, and LH only; we will miss the LL combination Note, that the actual order of the trials was randomized, but not shown here 18
CONVEYOR RESULTS The data from the 12 runs Visible Cracks Temperature Speed High Low High 3 3 4 3 3 4 Low 4 1 3 0 4 1 Using the ANOVA: Two-Factor with replication item from the Data Analysis menu yields the table on the next slide In this example, an eyeball analysis shows that we are probably onto something here 19
All three F-tests are significant, so we reject that there is no Speed nor Temperature effect We re pushing Excel at this point, but if we re-arrange the data, we can get it to give us an effects plot This is the place to run -> 20
A PICTURE S WORTH A THOUSAND WORDS When all the work is done and it s time to share the results, focus on the conclusions and use graphs that highlight the differences (or lack thereof) that you have discovered Avoid using statistical jargon when possible (e.g., say these data have a lot of variation rather than these data have a large variance ) Monetize your results whenever possible remember, executives think in terms of dollars, while engineers think in terms of things; you have to bridge the gap It s always good to have the statistical details in backup slides it shows you did your homework, and it will often discourage challenges make sure you can talk to them completely if you include them Make sure the experimentation team is all in agreement before presenting the results if you haven t convinced yourselves, how can you convince others? I like to write a short white paper in addition to the slide deck; it helps me rigorize the arguments, and it provides a more detailed documentation 21
7 STEPS TO SUCCESS WITH DESIGNED EXPERIMENTS* 1. Set objectives 2. Select process variables [factors] 3. Select an experimental design 4. Execute the design 5. Check that the data are consistent with the experimental assumptions 6. Analyze and interpret the results 7. Use/present the results (may lead to further runs or DOE's). * - From: The Engineering Statistics Handbook http://www.itl.nist.gov/div898/handbook/pri/section1/pri13.htm 22
ADDITIONAL RESOURCES We ve just scratched the surface on Experimental Design. Here are some additional references to take you to the next steps Anderson, Mark J. and Patrick J. Whitcomb (2015) DOE Simplified: Practical Tools for Effective Experimentation, Third Edition. Productivity Press. Excellent overview of experimental design Lots of good examples Includes a trial copy of Design-Ease, a full-feature DOE tool Wheeler, Donald J. (1990) Understanding Industrial Experimentation, Second Edition. SPC Press, Inc. Very thorough coverage of experimental design in an industrial setting Very practical Strong continual improvement focus Box, George E. P., J. Stuart Hunter and William G. Hunter (2005) Statistics for Experimenters: Design Innovation and Discovery, Second Edition. Wiley. The classic text on DOE for almost 40 years Strong theoretical coverage, but still very applied in the examples 23
24