How to Design Experiments

Transcription

1 September 14,

2 TABLE OF CONTENTS Lesson 1 - Experiments, Data, and Measurement The Experiment Data, Primary Data, Secondary Data Data: Quantitative, Qualitative, Discrete, and Continuous Measurement (Nominal, Ordinal, Interval, and Ratio) Relating Measurement to Qualitative and Quantitative Data Categorical Data Dirty and Clean Data Learning Tasks, Assessment Tasks, Portfolio Tasks 8 Lesson 2 - Authentic Science versus Pseudo Science Science (Authentic Science) Pseudo Science (Cargo Cult Science) Authentic Science Vs Pseudo Science Brain Flaws Changing from Pseudo Science to Science 11 Lesson 3 - The Experiment as a Method for Problem Solving 13 Lesson 4 - Experiment Design: Populations & Samples Populations and Samples Types of Studies Bias Probability Sampling Non Probability Sampling Learning Tasks, Assessment Tasks, Portfolio Tasks 19 Lesson 5 - Controlled Experiments Definition of a Controlled Experiment The Hypothesis Variables and the System Design Control Replication Single and Double Blind Experiment Blocking Multi-Variable Experiments Learning Tasks, Assessment Tasks, Portfolio Tasks 24 Lesson 6 - Scientific Measurement Validation Calibration Precision and Accuracy Selection of Instruments 25 Lesson 7 - The Standard Structure for Any Experiment The Template 27 Lesson 8 - How to Report an Experiment (The IMRAD standard) The IMRAD Standard Writing the Introduction Writing the Abstract The IMRAD Template Useful Facts Learning Tasks. 31 Lesson 9 - Learning Tasks, Assessment Tasks, Portfolio Tasks 32 September 14,

3 Lesson 1. Experiments, Data, and Measurement A great way to start learning about experiment design is build your vocabulary. This lesson will equip you to: Define experiment, data, primary data, secondary data; qualitative data, quantitative data, categorical data, clean data, and dirty data. Define measurement. Classify measurements as (a) nominal, (b) ordinal, (c) interval, and (d) ratio. Define categorical data. Define clean and dirty data The Experiment An experiment is a method for solving a problem that involves: Defining you problem (or formulating you question) Deciding that an experiment is the best method for solving this problem Designing the experiment (i.e., figuring out what data we are going to acquire) Collecting our primary data Drawing conclusions from the data Reporting your findings Examples Will my Design Idea work? Compare Design Idea A versus B. What is the head versus flow rate for water flow through coffee? (Characterizing a material) Will people want this product (better belay device)? (Identifying customer needs) How accurate is my math model (validating a math model) I want a model that relates the flow rate though a slow sand (building an empirical model) Rationale for learning ED: Fun (hands on), Data reveals the truth, Essential for validating math models, for design, for Facts. The method is not linear, instead it is recursive which means that you do the steps over and over until they have been done well enough to stop. I like the name research as in search and then do the search over. It is much like running through a corn maze. Doing an experiment can also be described as doing research or doing science. Doing science (i.e., designing experiments) is not at all like any lab class I ever had at a university. The process can be short (1 hr) to long (1 to 2+ years.). I generally recommend iteration cycles. September 14,

4 During this course, well add much more detail to the list of steps given above Data, Primary Data, Secondary Data In the context of an experiment, data refers to observation that we translate into (a) descriptions or (b) measurements. Observation Some useful facts about data are as follows. If you are recording the data (i.e., making the observations; taking the measurements), this is called primary data. If you are describing data recorded by someone else this data is called secondary data. Engineers talk about being data based. This means to base decisions on primary or secondary data (i.e., observations in the real world) as opposed to basing decisions on less useful things such as emotions, opinions, and conjecture. This course focuses on you taking the data (primary data) and you drawing your conclusions from this data. Data has multiple meanings. For example, data in the context of a computer programming mean information that can represented as zeros and ones. When you run an experiment, you collect data, and this collection is called a Dataset. One data point is a called a datum. The word data refers to multiple data points. Thus, the word data is plural. We write the word data as if it were spelled as datas. E.g. The data are validated. A parameter that you measure is called a variable. Example. If you build 10 rockets of a given design and measure the altitude reached by each of the ten rockets, the set of 10 measurement is your dataset. Your variable is the height attained which might be represented with the symbol h. September 14,

5 1.3. Data: Quantitative, Qualitative, Discrete, and Continuous Data are either quantitative or qualitative Quantitative Data: Uses numerical values to describe something that we observe. Data are measured on a numerical scale. Examples Third place in a bike race. Temperature inside a testing apparatus is 30 degrees C The angle at which the block slid was 22 degrees The data from the survey is 10 people said strongly agree or agree; 7 said no opinion; 15 said disagree or strongly disagree The times required for workers to process this part were (22.5, 19.1, 33.7, 15.0, 24.5) seconds Qualitative Data: Uses descriptive terms to characterize something that we observe. Data describe our observation. Examples. The surface of the water had waves in it. The flame at these conditions was red. The brand of car a person drives The number of students in a SCUBA diving class who complete the course versus the number who of students who do not complete the course. Quantitative data can be either discrete or continuous. Discrete data (or count data) can only take on certain values. Discrete data is counted. Examples: The number of heads in twenty flips of a coin, the number of students in a class, the number of faulty parts in 1000 parts off the assembly line. Continuous data can take on any values within a range. Continuous data is measured. Examples: The high temperature during the day, the weight of a dog, and the density of a material Measurement (Nominal, Ordinal, Interval, and Ratio) Measurement According to the international vocabulary of metrology (cite), measurement is the method that is used to assign a number to a characteristic (i.e., observation) of an entity (object or event), so that the given entity can be compared to another entity. In 1946 Stevens published an article in Science on the four types of measurement: nominal, September 14,

6 ordinal, interval, and ratio has been widely adopted. A nominal level involves qualitative data. Examples: Gender, colors, breed of dog, manufacturer of car. This data type does not allow any mathematical operations. Can be a simple description or a classification into a category An ordinal level involves rank ordering of data. Examples are as follows. Example. Place in race as in first place, second place, third place Example: Rating movies with 1 star, 2 star, 3 star or 4 stars Example: Likert scale is social science research (4 = Strongly agree, 3 = agree,.) Some facts about the ordinal level are as follow: Ordinal level data can be either qualitative data (e.g., Likert scale) or quantitative data (e.g., 5 star safety rating for cars) For ordinal data, you cannot compare measurements mathematically (e.g., a 4 star movie is not twice as good as a 2 star movie) An interval level provide information about order, it possess equal intervals, it applies to quantitative but not qualitative data, addition and subtraction are meaningful, but multiplication and division lack meaning. Example: A temperature of 20 oc is 5 oc warmer than a temperature of 15 oc. However, multiplication and division are not meaningful. Example. We cannot say that a temperature of 20 oc is twice as warm as 10oC because twice as warm does not really mean something. Example. Time of day on a 12 hour clock is an interval level measurement because (a) a time measurement is quantitative data (b) time-of-day has equal intervals; e.g., hour, (c) the intervals, e.g., hours, are equally spaced. (d) subtraction is meaningful because the difference between 2 pm and 6 pm is 4 hours, and (e) division is not meaningful because 6 pm divided by 3 pm lacks meaning. The level of measurement for a particular variable is defined by the highest category that the measurement achieves. For example, we can measure sound as loud or soft (nominal), we can measure sound as a 5 scale that spans from very soft to very loud (ordinal), we can measure sound in decibels (interval). Additional Reading. UC Davis website Relating Measurement to Qualitative and Quantitative Data September 14,

7 The image that follows show how the levels of measurement and the types of data are related Categorical Data Categorical data can be divided into groups or categories. Examples: race, sex, age group, educational level, mpg when divided into categories such as 0 to 10 mpg, 10 to 20 mpg, 20 to 30 mpg, > 30 mpg. Facts: Can be quantitative or qualitative 1.7. Dirty and Clean Data Clean Data. The data is accurate and only reflects noise that is really there. Free of bias and mistakes made by you (or me). Dirty Data. The data has issues, instruments have bias, calibration error experiment, there are mistakes by you or me. September 14,

8 The Dirty Data Rule. The data is always dirty (always has issues). We bend over backwards, as good scientists, to attain clean data but this is usually not possible. Only novices are sure that they have clean data and don t need to validate, etc. Data that been very carefully recorded (in general) will be not just dirty but really dirty and therefore not trustworthy. A good experimentalist is always high skeptical of their own data Learning Tasks, Assessment Tasks, Portfolio Tasks Here are some assessment tasks for this lesson. Here is the associated feedback for these ATs. For this lesson, there are not any learning tasks or portfolio tasks. September 14,

9 Lesson 2. Authentic Science versus Pseudo Science Now, do you want to do fake science? Or real science? Of course, all of us want to do real science. It turns out that the natural human tendency is to do fake science. This lesson will reveal this story and put you on a path towards real science. This lesson will equip you to Explain the differences and similarities between real science and fake science, (b) explain why most people (90 % perhaps) tend towards fake science, and (c) explain how to move towards real science as you method of choice Science (Authentic Science) Science (verb) is a method for figuring out the best or most correct ideas about the world. The method for doing science is to define the issue (problem or question) of interest design an experiment to get data to address your issue record your data draw conclusions by examine the data report your findings Throughout history, scientific breakthroughs have changed the world. The science of energy led to the steam engine. The science of flight led to the airplane. The sciences of microorganisms led to dramatically better medical care The science of silicon, electricity, and computing led to computers and smart phones Science (noun) is the result of the science; i.e., the knowledge that is presented by experts and by the best books, articles, and websites. Science tells you the best/most-correct ideas about the world; for example, science will tell you The healthiest ways to eat How solar cells work The most effective ways to train dogs The factors that effect the strength of an electromagnet The most effective ways to work in teams Some facts about science are as follows. Science address questions about the physical world; this is called physical science. What is measured are things like temperature, time, force, voltage, current, length Science addresses questions about people; this is called social science; what is measures are things like preferences, actions, responses, Science uses many methods Science is based on empirical evidence Scientific knowledge is revised when new/better evidence arises Scientific knowledge is built by communities The results of science (i.e., sci. Knowledge) explain how the world works. Examples NLs explain how motion works; Joule s CE explains how work and energy balance; Carnot s September 14,

10 theory tells us how much work we can get out of power plant Science is rare; what is mostly done is pseudo science 2.2. Pseudo Science (Cargo Cult Science) Richard Feynman, Nobel prize winner in physics, suggested in a commencement address that most people do fake science, an idea that he called cargo cult science. Cargo cult science (i.e., pseudo science) involves methods that have look like they are scientific methods, but they miss the essence of science and they lead to incorrect or misleading knowledge Authentic Science Vs Pseudo Science The first essential difference is the motivation of the person: Case 1 (sci). Motivation is to find out what is most likely true or what are the best ideas; intense curiosity; I just want to understand this Case 2 (pseudo-sci). Motivation is anything else. Examples. Prove that my idea is true. Get credit. Get a paper published. Prove that global warming is happening (or not happening). Protect a commercial interest (cigarettes, NFL helmets,.) The second essential difference is the beliefs of the person in the context of doing the work: Case 1 (sci). I am super worried about my methods. I want to bend over backwards to check my work. Case 2 (pseudo-sci). Certitude about methods. Anger when challenged. The third essential difference is the certainty: Case 1 (sci). The evidence suggests Case 2 (pseudo-sci). Certitude. Defensiveness. Table 1. A Comparison of Authentic Science and Pseudo Science Factor Authentic Science Pseudo Science Purpose Find out what is most true. Discovering the best ideas or the nature of something. Something else such as getting credit, proving something is correct, or providing data for selling. Ethical Approach Bend over backwards to validate and check your work. Be you own worst critic. Remove anything that might cause you to not achieve your purpose. September 14,

11 Certainty Humbleness. Certitude. Anger and defensiveness if challenged. Openess All methods revealed. Hides methods. Obscures. Reporting Report all results even those that might cause problems for you ==> describe truth Hide results that might cause problems for you. Based on many observations in industry and the university, I have concluded that it is human nature for people to do pseudo-science. It is rare for people to do authentic science. The reason has to do with brain flaws Brain Flaws Examples of Brain Flaws. Galileo versus the Catholic Church Rejection of the law of conservation of energy by scientists Rejection of plate technics Rejection of evolution Rejection of global warming Rejection of data by professionals A brain flaw is a natural way that the brain works that can sometime cause problems or harm to you (and potentially to people around you). The brain has evolved for survival and for reproduction. Anything that threatens the individual provokes an automatic defensive response. This defensive response is what causes pseudo science Changing from Pseudo Science to Science Traditional Mindset. I believe in scientific knowledge when it suit me. I reject any knowledge that I disagree with. I believe in the scientific method when it suits me. [My intuition is my guide: I have an inmate ability to know when I am right] Scientific Mindset. I believe in science knowledge, but I recognize that it can change in the face of new evidence. The sci methods is the best method for finding things out that humans have up with, but it may be possible to find a better method. Sci. has limitation and there is room for ethics, religion and other aspects of human culture. [I am never sure I m right because I am not sure what that even means ] Be skeptical, but try out science (get data as a way of reaching a conclusion). Use evidence. September 14,

12 Example 1: Math modeling Example 2: Training for bike racing. Example 3: Pr September 14,

13 Lesson 3. The Experiment as a Method for Problem Solving An experiment (or science) is a method for solving a problem by gathering data and then reaching conclusions from this data. Thus, being skilled at problem solving is a requirement for being skilled at designing experiments. I have a BookCourse on problem solving. This lesson introduces ideas from the problem solving course that essential for doing experimental work well. To avoid duplication, I have not copied the problem solving BookCourse into this BookCourse. Thus, please refer to this link for the content of this lesson. Your goals for this lesson are: 1. Define these terms: Problem, problem solving, ill-structured problem, and well-structured problem, problem formulation, goal state, present state, and rationale (in the context of problem solving). 2. Explain how to transform an ill-structured problem into a well-structured problem. 3. Explain the following three methods for simplifying hard problems: decomposition, simpler problem, and iteration. 4. Describe the standard methods for problem solving. Given a problem, select the method that is most appropriate for solving the problem. September 14,

14 Lesson 4. Experiment Design: Populations & Samples Once you have applied your problem solving skills to identify that a given subproblem is best solved by conducting an experimental study, you next design your experiment. This lesson gets you started. Your goals for this lesson are: 1. Define these terms: Population, sample, sampling, statistic, parameter, and inference. 2. Describe the following methods of data collection (a) census, (b) sample survey, (c) qualitative methods, (d) controlled experiment, and (e) observational study. 3. Define bias and describe some of the common types of bias. 4. Know how to apply each of the following sampling methods: (a) simple random sample, (b) systematic sampling, (c) stratified sampling, (d) cluster sampling, and (e) multistage sampling. 5. Define convenience sampling Populations and Samples Population. Your population is all the entities (people, widgets, ) you want to know something about. Examples: I want to know the actual voltage (i.e., not the rated voltage) of every AA battery manufactured by my company. I want to know the opinion of every employee who works for my company. For most experiments, it is not possible to gather data from every entity in your population. Example: If you worked at Boeing and you want to gather data from every employee, this would require getting data from about 162,000 people. Example: If you are designing a product and you want to gather data from potential customers, you don t even know who these customers will be, much less how to gather data about them. Example: If you want to gather data about the strength of a part that your company makes, you will need to measure the load required to break the part. If you wanted to test every part, you would need to break every part made which is obviously not practical. People before us have figured out how to deal with this issue. The methods involves 1. Select a few of the entities from your population. This limited set is called a sample. The method for selecting this set is called sampling. 2. Gather data from this limited set. This is your experiment. 3. From your experiment involving a sample, make conclusions about your population. These conclusions are often called inferences. September 14,

15 A number that describes a population is called a parameter. A number that describes a sample is called a statistic. Different symbol are used for statistics and for parameters. For example, the mean of a population is given the symbol μ and the mean of a sample is given the symbol y. Fig. 1 shows how a population and sample are related. Figure 1. A sample is a subset of a population. When we use a number such as the mean to describe a sample, this number is called a statistic. When we use the same parameter to describe a sample, this number is called a parameter. [this figure was copied from the internet; will be modified/improved to avoid copyright issues] Some other useful images are shown below 4.2. Types of Studies When you go to do your experiment, you select the type of experiment (i.e., type of study) you are September 14,

16 going to do (Table XX). Table XX. The Three Types of Experimental Methods. Relevance refers to how important this method is to people who focus on technology development. Name and Description Relevance Physical Social Science Science Census. You take data on every entity in the population. A low yes yes census is usually not possible because of cost and other practical matters. Controlled Experiment. You divide the entities into high yes yes appropriate groups and impose some actions (i.e. treatments) on some of the groups. You generally define a control group and one or more treatment group. The next lesson goes into detail on this method. Observational Study. You observe and record data but you do not impose any kind of treatment. The three main types of observational studies are described below. Qualitative Methods. You record data about people by high no yes methods such as interviewing selected members of your population. You generally do not sample because you want preliminary data. Johnson provides an in-depth discussion. Sample Survey. You select a sample of your population low no yes and use a survey to acquire data from the sample. Scientific Observational Study. You observe scientific data (not social science data) in the field. high yes no Fact. A longitudinal study is a study done over time Bias Bias is something that may cause your results or conclusions to differ from the true results or true conclusions. Of course, you do not know the truth, so there is no way to detect bias. All you can say is that certain methods introduce bias into your experiment. Your aim is select methods that reduce and eliminate bias. Facts Sampling bias refers to errors caused because you did not sample properly. Measurement Bias (in measurement) means that one or more of your measuring instrument is not working correctly and all your measurements are off because of this. Researcher bias refs to There are many types of bias in surveys: September 14,

17 Non-response bias. People do not answer the survey. This may cause bias because the people who choose not to respond may be different (as a group) than those who choose to respond. Selection bias. People self-select to respond to the survey. This may cause bias because people with strong opinions are the ones who will self-select. Undercoverage bias occurs when part of the population is ignore; for example an survey will only reach those people who use and will not reach people who either do not use computer or those who ignore . Response bias is caused by the wording of questions. People often don t want to answer a question in a way that might suggest that they are unpopular, they have unethical views, etc. Big. Idea ==> Statistical methods cannot fix bias. Bias causes dirty data. As a skilled researcher, you strive to eliminate bias. This is often not easy Probability Sampling The goal of sampling is to produce a representative sample which means a sample that has the essential characteristics of the population and that avoids any type of systematic bias. The sampling frame is a list of all the units of a population from which you or I can draw a sample. A probability sample is one in which each unit in the population has a known probability of being selected and you use random mechanisms to select the units that form your sample. Note that the probability of being selected does not need to be the same for each unit. A Random Sample means that every member of the population is equally likely to be chosen. There are four common methods for taking probability samples, these four methods are described in the text that follows. It is also common to combine several of these methods together; this is called multistage sampling. Method #1. Simple random sampling (SRS; Video) is a method of sampling in which all possible sample samples of n elements are equally likely to occur. (here, the population has N elements and the sample size is n). A sample can be a random sample but not a SRS. The classic example is selecting at random 2 members from each NFL football team (this is a random sample but not a SRS). Method #1. Systematic sampling (Video) means that the first unit of the sample is chosen by a random method and then the rest are chosen according to a well-defined pattern. Example. You use a random number generator to decide to pick the 7 th person who enters a store. Then, you select every 10 th person. September 14,

18 Method #3. Stratified sampling (Video) involves dividing your population into subgroups called strata. Then, you sample out of each strata. The members of a given strata are alike in some way. Example: you create a male strata and a female strata and the you apply SRS to select 25 males and 25 males for your sample. Method #4. Cluster sampling (Video) involves dividing your population into subgroups called clusters. A cluster involves a group that is already in place. For example, a cluster might be all the people in a city (a city is already in place). A cluster might be all the parts manufactured by a given manufacturing facility (the facility is already in place). This is in contrast to stratified sampling in which you (the researcher) define your strata. In general, you want a cluster to be representative of your population; we say that the cluster is heterogeneous (diverse). This means, for example, that a city selected as a cluster tends to have people in the city who are reflect the diversity of the all the people in your population. Cluster sampling involve selecting one or more of your clusters as your sample Non Probability Sampling Non probability sampling is a type of unit sampling in which (a) some units have a zero probability of being selected, and (b) the probability of a unit being selected is unknown. Read more: Non-probability sampling is to be avoided if you are striving to reach conclusions about your population. Non-probability sampling is useful if you are developing your methods or in cases where you lack knowledge of your population. For example, if you are developing a new product, it is common that you are not sure who your target customers are and you need to sample people to establish who is most likely to buy your product. Some of the methods for non-probability sampling follow. A self-selected sample (voluntary response sample) describes a group of people who volunteer to participate in your study. For example, a web-based survey in which people choose to respond (or not) is a self-selected study. A self-selected sample of people tends to result in a sample of those people with strong opinions; this self-selected samples tend to cause sample bias. A convenience sample describes a study in which you find the units of your sample in a way that is easy for you. For example, if you are going to measure the failure strength of bolts and so you go September 14,

19 to the nearest hardware store and buy a box of bolts, all from the same manufacturer. This is convenient for you but may not result in a valid sample. Thus, you run the risk of drawing incorrect conclusions from your data when you use a convenience sample Learning Tasks, Assessment Tasks, Portfolio Tasks Learning Task. To learn about experiment design, watch this video (part 1) from Stat Trek. Then, watch this video (part 2). The notes from the videos are found on this page (part 1) and this page (part 2). September 14,

20 Lesson 5. Controlled Experiments In the context of technology innovation, a controlled experiment is used to figure out how to make your technology better. Since, making great products is at the heart of technology innovation, this lesson is highly relevant Definition of a Controlled Experiment You divide the units under test into appropriate groups and impose some actions (i.e. treatments) on some of the groups. In general, you will put some units in a control group (nothing happens to them) and some units into a treatment group. The main purpose of a controlled experiment is to determine cause and effect. Effect typically is what happens. Cause describes the factors that lead to this result. That is, cause leads to effect as in (cause effect). Cause and effect is the essential thing that the technology innovator want to determine. Some examples are given the paragraphs that follow. Suppose you are designing a new model car for Ford. If you can figure out cause and effect, you can improve the gas mileage. In particular, you figure out which variables cause the gas mileage to increase and then you can redesign the car by changing these variable so that you get a car that attains better gas mileage. You can also save lives. If you can figure out which variables cause the car to better protect the passengers in collision, then you can redesign the car by changing these variables so that the passengers are safer in collisions. Suppose you are designing a new technology product. If you can figure out which variables cause customers to buy this product, then you are on your way to a product that succeeds in the business world (i.e., the product is financially viable). Suppose you want to design a new football helmet that reduces the number of conclusions. If you can figure out which variables cause the helmet to be safer, then you can redesign the helmet and thereby make your product much better for athletes The Hypothesis A hypothesis is label for what you expect to happen in your experiment. For example, I expect that if I increase the wall thickness on this part by 15%, then the defection will go down by a September 14,

21 factor of 2. Not all experiments have hypotheses. Hypotheses are fun, especially when my guess is proven wrong because I can then learn something. Hypotheses in engineering are especially useful when they are based on math models Variables and the System Design Skillful research begins with understanding variables. A variable or random variable is something that you measure and something that changes during different trials of your experiment. An independent variable (explanatory variable, factor, input) is a variable that you change the value of during your experiment. The values of your IV are sometimes called levels. A dependent variable (response variable, output variable) is a variable that you observe. This variable often changes in response to changes in your In general, to confound means to mix up (something) with something else so that the individual elements become difficult to distinguish. A confounding variable is a variable that is not one of the independent variables (treatment variables), yet it has an influence on the dependent variable. Thus, you cannot separate the effect of the confounding variable from that of the independent variables. A lurking variable is a variable that is not included as an explanatory or response variable in the analysis but can affect the interpretation of relationships between variables. A lurking variable can falsely identify a strong relationship between variables or it can hide the true relationship. According to this webpage, confounding and lurking variables differ because a confounding variable is known to you and you eliminate the influence this variable by skillful design. A lurking variable is one that you either are not aware of or you can not do anything about; it lurks beyond your influence. Your system is whatever you want to study. A system diagram is a sketch (fig xx) that shows your system, your input variables (explanatory variables), your output variables (response variables) and your extra variables. Extra variables are defined as all other variables that you need to consider but that you are not varying in the context of your experiment. September 14,

22 For example, if you are doing a study to determine how long a solar cell takes to charge a battery, you explanatory variables might by parameters such as x = (length of the solar cell, width of the solar cell, angle of the solar cell with respect to the sun, the solar irradiation, and brand of solar cell). Your response variables might be parameters such as y = (voltage at time t, time ). Your extra variables might be parameters like z = (shading from objects like trees, time of day, air temperature). Some web images that I need to redraw follow this paragraph Control September 14,

23 In an experiment, control means holding all variables constant except for the treatment variables (independent variables). Control is essential because these prevents unwanted variables from influencing your results Replication Replication involves multiple trials of an experiment. You repeat the experiment with different units, at different times, etc Single and Double Blind Experiment In a single blind experiment, the experimental subjects do not know if they are part of an experimental control group of a group receiving a treatment. The purpose of this is to prevent bias on the part of the subjects In an experiment, subjects respond differently after they receive a treatment, even if the treatment has no "real" effect on the dependent variable. Such a treatment is called a placebo, and a subject's positive response to a placebo is called the placebo effect. In a double blind experiment, neither the experimental subject or you (the researcher) know if a given subject is part of an experimental control group or part of a treatment group. The purpose of this is to prevent bias on the part of the subjects and bias from you (the researcher). You apply a treatment to some units in your experiment and you do not apply a treatment to other units. If there is a difference in the dependent variables between units without treatment and this with treatment, this suggests that the independent variable have some effect. The group of units without treatment is called the control group Blocking Blocking is the arranging of experimental units in groups (blocks) that are similar to one another. The purpose is control for the effect of a confounding variable. Blocking is applied to eliminate the effect of a variable that is not of interest to you. This website provides a thorough explanation of blocking. September 14,

24 5.8. Multi-Variable Experiments Common sense, for most people, is to change one variable while holding all others constants. This allows you to see how your selected variable effects the results you want. After you see the effect of variable A, you select a different variable B and change this variable. However, this common sense approach can lead to dirty data because you will miss interactions between variables. It is also not suitable to many real world problems because you might face 5 to 50 independent variables. Fortunately, researchers have figuring out a much better technique in which you can vary, for example, 7 variables at once and get data that tells you the relative importance of all 7 variables and the degree to which the 7 variables are interacting Learning Tasks, Assessment Tasks, Portfolio Tasks Learning Task (repeated from the previous lesson). To learn about experiment design, watch this video (part 1) from Stat Trek. Then, watch this video (part 2). The notes from the videos are found on this page (part 1) and this page (part 2). Assessment Task. This word document (from the internet) has assessment tasks that are essential. I have not yet built the feedback. Lesson 6. Scientific Measurement As a technology innovator, many of your measurements will be made with scientific instruments such as force transducers, thermal couples, voltage meters, oscilloscopes, and pressure measuring instruments Validation To validate means to run checks that provide evidence that your measurements can be trusted. Examples: To check your temperature measurement system, you measure the temperature of an icewater bath to make sure that it is 0 degrees C. To check your force measurement system in a wind tunnel, you measure the drag force on a sphere and then check this with a calculation that you make using reference data from your fluid mechanics book. September 14,

25 Triangulation involves three independent validations of your measurement system. If all three validations tell you that your system is accurate, then you can have some level of confidence Calibration 6.3. Precision and Accuracy Accuracy. If you take the mean of your measurements, the number is close to the true measurement. Precision. Your measurements are very repeatable meaning that the standard deviation of your dataset is very small compared to your mean. Example. Your measurements of voltage are (12.01, 11.98, 12.03, 12.05, 12.02, 12.00, 11.99) volts. A precise measure is not necessarily accurate because of measurement bias. Facts and ideas: Precise but not accurate is common in professional practice. For preliminary experiments, I recommend accurate but not precise because this saves time and money For high stakes experiments, you may need accurate and precise. Always focus on accurate first. The image below captures the ideas Selection of Instruments Big ideas. Select the simplest, lowest cost instrument that gets the job done. Select non-electronic instruments when possible. Avoid complex high-tech system unless they are justified (these September 14,

26 systems have a high vacuum effect as in they suck you in because they are super cool technology). Avoid automated data acquisition systems unless you have reached the limits of hand operated. Figure out how your instruments work or suffer my fate you will (losing months and years of time due to dirty data). September 14,

27 Lesson 7. The Standard Structure for Any Experiment Nearly every experiment that you do or that I do can be done well if we attend to certain fundamentals. Alternatively, if we miss a fundamental, the experiment will likely to be lousy. The purpose of this lesson is to organize the fundamentals so that your experiment are nearly always well done The Template Big idea. The experiment process is best done iteratively. I recommend three iterations. The report itself may take more iterations. Step 1. Define your problem Step 2. Define your experiment Step 3. Record your dataset Step 4. Troubleshoot and loop back to Step 1. In parallel with steps 1 to 4, write drafts of you project. Define the Problem. What is the issue? What is the goal state? Why care? Why is an experiment the best way to address the issue? Could you address the issue via on of the other standard problem solving methods? September 14,

28 September 14,

29 Lesson 8. How to Report an Experiment (The IMRAD standard) When you write a report describing an experiment, there is a standard recipe that is followed in most areas of science and engineering. Following this standard makes your report much easier to write and much easier to read. This lesson will equip you to (a) Describe the IMRAD standard, (b) List or explain what goes in each section of a report of an experiment, (c) Write a report following the IMRAD standard The IMRAD Standard The IMRaD structure is shown in Figure xx. The parts of the report are as follows. Introduction: Why was this project done? So what, why does this matter and who cares? Method: How was this project done? Results: What did you find out? Discussion: What do your findings mean? What next? Conclusion: What are the main results that you want to provide? Figure xx. The IMRaD structure can be described using the Wineglass model. The top (i.e., the introduction) defines the problem you are trying to solve. The bottom (discussion/conclusion) presents your solution. The middle (methods/data) explains your methods and how you processed your data. The abstract (top) summarizes the parts of your report. [image adapted from Wikipedia] Some facts about the IMRAD structure. September 14,

30 8.2. Writing the Introduction The purpose of the introduction is to define the problem. Tell the reader what 8.3. Writing the Abstract An abstract of your report is a summary of your report that is typically one or two paragraphs long and one hundred to three hundred words. The information that follows explain how to write an abstract and also describes some elements of quality. Quality in the context of doing something (e.g., writing an abstract) means doing things in the same ways that skilled professionals do things. Example. Quality in playing the guitar means doing things in ways that are similar to what all skilled guitar players do. The Four Functions of the Abstract Rationale/Goal. Tell the reader why you did the research, what problem you are trying to solve, and why solving this problem matters. [optional: tell the reader what your goals were or what your research questions was] Methods. Tell the reader how you solved the problem. Quality. Specific (not vague). Reproducible (a skilled person can replicate your methods) Results. Tell your reader what you discovered. Quality. Specific. Humble. Limited to the scope of what you did (avoid over generalizing). Implications. Tell your reader what your results mean and why they matter. [optional: suggest next steps] The IMRAD Template 8.5. Useful Facts Your report can informal or formal, but follow the IMRAD standard. Use SI units. Justify the # of significant figures. If you use more than 3 SFs, you had better be able to explain why this is justified. September 14,

31 8.6. Learning Tasks. Watch this video in which I explain how to write a scientific paper. Read this research paper about the IMRAD structure. Notice how the paper applies the IMRaD Read Richard Jewell s article on the IMRAD structure. September 14,

32 Lesson 9. Learning Tasks, Assessment Tasks, Portfolio Tasks September 3, 2015 To see current Assessment Tasks, click here. To see current ATs with feedback, click here. September 10, 2015: ATs on experiment design vocabulary. September 14,

33 9.1. The Funnel Project (A Portfolio Task) Overview A technology company headquartered in China has outsourced a project to an American consulting firm. They have outsourced this project to you. The company needs to predict the time for a tank of room temperature to drain Your task is to build a suitable math model and then to perform an experimental study in which you quantify how well a math model can predict data from the physical world. The tanks of interest are shaped basically like a funnel. Requirements. Solve the problem and deliver a report. IMRAD standard. Follow the ASME Fluids Engineering Division Format. Work as a team of two (self-select). Due Thursday beginning of class; bring 3 copies; each printed and stapled; peer assessment will be used. late penalty = 25% of your points. September 14,

34 Grading The rubric (from the syllabus) is given below. In the context of professional performance, quality means that you take actions that would be judged as sound by experienced professionals who are skilled with the performance in question. Examples: Quality in playing guitar involves actions such as (a) holding the rhythm, (b) playing in tune, and (c) playing notes and chords that make a pleasant sound. Quality in computer programming involves actions such as (a) creating code that runs without any errors, (b) documenting code well, (c) following conventions associated with the the programming language you are using, and (d) appropriate use of libraries. Summary. Quality in performance involves taking the appropriate actions to produce results that are valuable to other people. Appropriate actions are those actions that most (80% or more) skilled professionals take. Level 1 Quality Standards (for this project). Your report is submitted on time. Your bring the requisite number of copies to class. Your organization reflects the IMRAD standard. Your report follows the ASME Fluids Engineering Division Format Level 2 Quality Standards (for this project). You meet all the requirements for Level 1 plus most of the following. You provide your reader with results that are useful. You define the problem that you are striving to solve. You review the existing knowledge and connect it to this project. You explain the steps need to derive your your math model from basic equations. You explain your experiment methods well enough so that any researcher could replicate your methods. Your writing reflects the essence of science (humbleness + search for the truth). You apply appropriate methods and methods from statistics. September 14,

35 Class Discussion Topics Goal State?. To find your goal state for anything ask the question, what will extraordinary success look like and feel like to me (us)? Focus on yourself (what you want) and on your client. Avoid focusing on your manager or your teacher. Hard to Easy ==> How? The big three strategies are (1) decomp, (2) simpler problem, and (3) iteration. Decamp. Break into math model, experiment, write report,. Simpler problem. Build simplest possible math model. Build simplest possible experiment (funnel from WalMart) Iteration. Do early experiments. Immediately compare with math model. Repeat. Scaling (large tanks) Apply Dimensional Analysis from fluid mechanics. Conclusion: Use small tanks that are geometrically similar. Since the flow involves converging streamlines, can model w/ inviscid flow; thus the Reynolds number does not matter. Conclusion: You or I can make a sound/ strong argument that you can test with models. Math model. Will be done in class. Population? Sample?. Pop = collection of all designs that might be built; sample = 2 to 6 experiments with varied geometry. I d probably used stratified sampling. How to build a tank. Buy or build a low cost funnel. Measure time to drain. Compare data with prediction of your math. Do a diamond to find many simple ways to build a tank. (paper filter from coffee, paper filter from where I buy oil). September 14,

36 Class Topics (September 15, 2015) Class Topics (September 15, 2015) Writing the Introduction. Will be done in class. Key ideas. Problem Goal + Rationale. See yourself as a problem solver. Start writing up your report on day #1. Applying Statistics. Will be done in class. Key idea: Measure time to drain (variable #1). Observe the time to drain as predicted by your math model (variable #2). Form a new variable which is (t_measured - t_predicted)/t_measured (variable #3). Build a small-sample confidence interval for variable #3; the formula (from p. 323 of your text) is ( ) x ± t c s / n Control of your experiment. Will be done in class. Key idea: design a systematic method for doing your experiment so that a given replicate of your experiment will give a consistent drain time. September 14,