Ratios and Proportional Reasoning. Assignment #2. Erica Sautter. Marist College

Transcription

1 Ratios and Proportional Reasoning Assignment #2 Erica Sautter Marist College

2 Sautter 2 Abstract One of a teacher s primary responsibilities is to ensure that their students are meeting Common Core State Standards through the lessons they teach. Completely covering each one of these standards is not something that happens in one forty minute period. It is something that takes days, weeks, and even months sometimes to cover completely. In order to be sure students gain the knowledge expected of them set by the standards before the end of the school year, one tool many teachers use is known as a test blueprint. This paper will follow a seventh grade mathematics teacher through the process of creating a test blueprint for a unit on proportional reasoning. This blueprint will include the three major contents in this unit including: ratios, proportions, and applying percents. The teacher will then discuss eight blueprint assessment points providing a variety of informal and formal formative assessments that could be used as well as a description as to why these points were chosen based upon the test blueprint created. Finally, the teacher will reflect on the validity and reliability of the test blueprint and assessments.

3 Sautter 3 My Class The class I will be teaching this unit to will be a regular seventh grade class that has nineteen students, eleven boys and eight girls. Fifteen of my students are Caucasian, three students are Chinese, and one student is Hispanic. Two students have ADHD, one student has Asperger s, and two students have IEPs. Additionally, sixty percent of my class qualifies for free or reduced lunch. For one of my Chinese students and my one Hispanic student, English is a second language. The Chinese student is at the beginning stages of learning English while my Hispanic student is at the intermediate level. Both students have a hard time grasping concepts when I lecture for long amounts of time as they both prefer hands on and visual activities. If I do need to give a direct lecture for the majority of the period I will hand these two students completed notes so they can spend more time understanding the concepts presented rather than worrying about writing it down. I have found they tend to have a greater understanding of the material when it is something they have discovered while working with fellow classmates or seen presented visually on the board. Luckily in mathematics, I do not give long readings for students because if I did I would definitely have to give these two students extended time to finish the reading as it takes them longer to retain the information presented. Similarly, they would be granted extended time for a short writing response I give them in class. They would have this extra time, if needed, as it takes them longer to generate their thoughts and present them on paper. Two students in my class have IEP s for a variety of learning disorders including speech language impairment, a learning disability in math and reading, and processing speed. Additionally I have two students with ADHD and one with Asperger s. For these students I will provide a variety of modifications including frequent checks for understanding through oral

4 Sautter 4 questioning or observation, a copy of class notes when I see they have not taken notes of good quality, repetition of directions, and special seating. For special seating arrangements I will try to keep these students as close to the front and center of the class as I can as it will make it easier and less obvious for me to check their understanding of the topic throughout the period. Additionally I will try my best to keep the class moving as to not lose the focus of any one of these students. When possible I will incorporate stations in my class so that everyone can be up and moving so I can help any students who need extra assistance learning the material. The Unit All of the students in this class make up a seventh grade mathematics class who will be studying a unit on proportional reasoning. Throughout this unit I will be focusing on several Common Core State Standards that are listed in the seventh grade mathematics curriculum. These standards include the following: 7.RP.1 Compute unit rates associated with ratios of fractions, including ratios of lengths, areas and other quantities measured in like or different units 7. RP.2 Recognize and represent proportional relationships between quantities. a. Decide whether two quantities are in a proportional relationship, e.g., by testing for equivalent ratios in a table or graphing on a coordinate plane and observing whether the graph is a straight line through the origin. b. Identify the constant of proportionality (unit rate) in tables, graphs, equations, diagrams, and verbal descriptions of proportional relationships. c. Represent proportional relationships by equations.

5 Sautter 5 d. Explain what a point (x.y) on the graph of a proportional relationship means in terms of the situation, with special attention to the point (0,0) and (1,r) where r is the unit rate. 7.RP.3 Use proportional relationships to solve multistep ratio and percent problems. 7.G.1 Solve problems involving scale drawings of geometric figures, including computing actual lengths and areas from a scale drawing and reproducing a scale drawing at a different scale. I have broken this unit up into three different content areas that I feel are important for students to learn. The first content area is ratios. During this part of the unit students will learn what a ratio is and why ratios are important in everyday life. The second content area is proportions. During this part of the unit students will use what they learned about ratios to create proportions. These proportions will assist them in solving real life problems while allowing them to create proportional relationships of their own. The final content area of the unit is applying percents. Here students will apply what they know about ratios and proportions to solve real-life problems that include items such as tax, tip, and discounts. Throughout the entire unit students will learn to apply what they learn about these concepts to the real world as this unit is full of real world applications. These three content areas: ratios, proportions, and applying percents are shown in my test blueprint below. My test blueprint breaks down my goals for the unit based on Bloom s taxonomy and the Common Core State Standards aligned with this unit. I feel that each content area is almost equally as important as the next, hence why all three areas are around the same percentages. I felt that the proportions content area was a little more important as it relates to both ratios and applying percents and that is why it is weighted a little more heavily than ratios

6 Sautter 6 and applying percents. The levels of Bloom s taxonomy on the other hand are weighted based upon what the Common Core State Standards say should be covered and how important I felt each aspect was in meeting those standards. Based upon my percentages, it is obvious that both I and the standards feel it is important that students be able to analyze and create problems/ideas based on the content covered in the unit hence why they are weighted more heavily. This test blueprint will assist me in creating formative assessments throughout the unit to measure student learning prior to administering a summative assessment. These formative assessments will assist me in making decisions on when it is okay to move on in the material and when we may need to spend an extra day practicing a concept before moving forward.

7 Sautter 7 Ratios and Proportions Test Blueprint Proportional Reasoning Unit Ratios Remember Understand Apply Analyze Evaluate Create Totals Students will define what it means to have a ratio. 3/ 3% Students will explain the purpose of a rate. 2/ 2% Students will write real world ratios. 5/ 5% Given a table, graph, verbal description, or chart students will identify the rate shown. Students will evaluate ratio/rate problems. 10/ 10% Students will create a ratio/rate problem of their own to solve. 5/ 5% 33 / 33% Proportions Students will describe the similarities between creating a ratio and creating a proportion. Students explain how they create proportions based upon relationships presented. Students will write real world problems that require proportions to be solved. 8/ 8% Students will describe the proportional relationships displayed in tables, graphs, & verbal descriptions. Students will justify how they used proportions to solve problems. Students will create their own map scale project using proportions to find model and actual distances. 39/ 39% 5/ 5% 3/ 3% 5/ 5% 10/ 10% 4/ 4% 12/ 12% Applying Percent Students will recall how to convert between fractions, decimals, and percents. 3/ 3% Students will explain their thought processes when finding percentages with or without a proportion. 5/ 5% Students will demonstrate how they think about using percents in real life in relation to tax, tip, discounts, interest, etc. 3/ 3% Students will use given information to solve for percentages and other missing information. 10/ 10% Students will assess when it is appropriate to use a proportion and when they should use the percent equation to solve problems. Students will create their own situations to determine items such as tax, tip, discounts, interest, commission, etc. 5/ 5% 28/ 28% 2/ 2% Totals 11/ 11% 10/ 10% 13/ 13% 28/ 28% 16/ 16% 32/ 32% 100/ 100%

8 Sautter 8 Justification for Blueprint This section will focus on each of the eight points in my test blueprint that I have chosen to designate as points where I will incorporate formative assessments into my lessons to ensure student understanding. Each of these points is marked in blue in my test blueprint above. For each point I will describe why I chose that point, what standard the point addresses and one way I will gather formative assessment. Content Area: Ratios Cognitive Level: Remember: Students will define what is means to have a ratio. Common Core State Standard: 7.RP.1 Compute unit rates associated with ratios of fractions, including ratios of lengths, areas and other quantities measured in like or different units. Prior to teaching this unit student s will most likely have heard or even used the term ratio previously. However what they may not know or realize is how this term is properly used and what it actually represents. I felt it was important to be sure that all students understand right from the start what a ratio is as it is a term that will appear numerous times throughout the unit. If students are unsure what a ratio is by itself then how will they know or be able to realize and describe its usefulness when creating proportions? I plan on introducing this unit with a KWL chart (see Appendix A). Students will be asked in the beginning of class to take a few minutes to fill in what they know about ratios on their own. After they have had three minutes to write down their thoughts we will create a class list on the board. As the students fill in the what they know column I will walk around the

9 Sautter 9 room to observe the types of things they are writing down. I would expect them to come up with ideas such as something that compares two things, a fraction, and maybe even an example. I will guide these thoughts by providing a few guided questions in the KWL chart. For example, under the Know column I will include questions such as where have you seen ratios used, what is a ratio, and an example of a ratio is By the end of the period I will be able to gauge student understanding of the concept as I give them another three minutes to answer the questions we created as a class to answer throughout the period. As they take the time to fill in the column on what they have learned, I will again walk around the room assessing their understanding of the concept. If I see that the majority of students are having difficulty with the concept then I will know that the next day I need to reinforce the concept before moving forward. Prior to the end of class I will create a class list of the ideas that should be placed in the learned column using student input. I chose to use a KWL chart for this test blueprint assessment point because I feel it is something that students will be able to discover on their own using their previous knowledge. I feel this method is a thousand times better than when I just use direct instruction. If they are able to relate this term to something they know then they will be able to remember it a lot better than they would if they were just told what they term is and means. Cognitive Level: Analyze: Given a table, graph, verbal description, or chart students will identify the rate shown. Common Core State Standard: 7. RP.2 Recognize and represent proportional relationships between quantities. b. Identify the constant of proportionality (unit rate) in tables, graphs, equations, diagrams, and verbal descriptions of proportional relationships.

10 Sautter 10 I chose this blueprint assessment point because I feel it is important that students are able to look at any type of information and be able to conclude if the relationship shown is proportional or not and if it is what they constant unit rate is. Not to mention the standards explicitly state that students need to recognize and represent proportional relationships between quantities. It will not be enough for students to only know how to look at a graph and tell if the relationship is proportional, they must also know how to look at a table of data and be able to conclude if that relationship is proportional. If students are able to go from graphs to equations to diagrams to verbal descriptions to tables and conclude whether the relationship is proportional then that proves they have a strong understanding of the material being presented. I plan on assessing this point using a short quiz (see Appendix B). This quiz will be a four question quiz that will ask students to look at tables, graphs, verbal descriptions, and equations to decide if the relationship shown is proportional or not. If the data is proportional they will be asked to find the unit rate. This short quiz will allow me to formally assess each student s knowledge on proportionality. By using all formats the data could be presented in I will be able to gauge which formats students understand well and which ones are still questionable. This will also allow me to fix any misconceptions students may have as I give them feedback on the quiz. I chose a quiz for this assessment point because I feel it is very important that students don t move on in the content with any major misconceptions. By using a quiz I can assess each student s level of understanding and address any major misconceptions I see from several students. If my students excel on this quiz then I know they are on their way to meeting the goals set forth by the standards they are expected to know by the end of the unit. Additionally, for

11 Sautter 11 students who are struggling a bit I will be able to scaffold my lessons to meet their needs and hopefully present the material in a different way that they may be able to grasp. Cognitive Level: Evaluate: Students will evaluate ratio/rate problems. Common Core State Standard: 7.RP.1 Compute unit rates associated with ratios of fractions, including ratios of lengths, areas and other quantities measured in like or different units. I chose this point as one of my blueprint assessment points because I feel it is important that students know not only how to determine if a relationship is proportional but also be able to determine the rate or unit rate described in the relationship. Additionally this is a goal that the Common Core State Standards say all seventh grade students should be able to achieve. The way I plan on assessing this point is with the assistance of white boards. In my classroom we have a class set of personal whiteboards for the students to use. Having a dry erase marker is part of each students required materials for class. This method will allow me to pose questions on the board and each student can respond in their own manner on their white board. Once they have written down what they believe is the right answer they can hold up their board so that I can quickly assess their answer and provide feedback if they made an error in their calculation. If a student gets the right answer I am able to ask them to assist their neighbor who may be struggling. This type of assessment is very valuable as I am able to hold every student accountable for creating a solution to every problem. I can mark my seating chart to show who answered what questions and which students needed extra assistance before developing the correct answer. This also allows me to quickly assess my students and find misconceptions many of them have in a short time period.

12 Sautter 12 I chose to use white boards for this assessment point, because finding unit rates is something that can be determined fairly quickly. Therefore I will be able to go through numerous types of questions with my students. Also it will be helpful to those who may still be struggling as they will be provided with frequent feedback, more so than if they were working in small groups. Content Area: Proportions Cognitive Level: Analyze: Students will describe the proportional relationships displayed in tables, graphs, & verbal descriptions. Common Core State Standard: 7. RP.2 Recognize and represent proportional relationships between quantities. A. Decide whether two quantities are in a proportional relationship, e.g., by testing for equivalent ratios in a table or graphing on a coordinate plane and observing whether the graph is a straight line through the origin. D. Explain what a point (x.y) on the graph of a proportional relationship means in terms of the situation, with special attention to the point (0,0) and (1,r) where r is the unit rate. Prior to this unit students will not know that points such as (1, r) represent the unit rate of a graphed line. Therefore I felt this assessment point was beneficial as it would allow my students to discover the importance of certain points on a graph. In order for a line to be deemed proportional it must pass through the origin or point (0,0). Therefore the origin is a major indicator if a line is in fact proportional. More importantly, if a line is proportional then the point (1, r) represents the unit rate of the line. These two points are major ideas that the students must come to understand. If students can look at a graph or table and categorize it as proportional or

13 Sautter 13 non-proportional, then they have met the first major part of the goals listed by the Common Core State Standards. For this assessment point I decided to give my students a homework assignment (see Appendix C) that would show them three graphs and three tables. For each graph and table I would ask them to write a few sentences describing the relationship represented. If the relationship was proportional then I would ask them to include the relationships unit rate as well as justification for one of the points on the graph, stating what it is actually representing in terms of the labels on the graph. If students are able to determine if a graph or table is proportional and what the unit rate is then I know they have met this standards goal. The day the homework assignment is due I will have students come into class and compare their answers with their neighbors. While they are comparing answers I will walk around to check their completed homework looking for any major problems. Once I have finished checking the homework, I will ask if there are any questions that could not be solved after talking with their neighbor. I chose to give a homework assignment to assess this blueprint point because then students were able to sit back and really try to answer these questions on their own without the help of a classmate. If students were able to answer the questions outside of class fully and completely then they must have a strong understanding of the concept. This also allowed for extended practice of the concepts covered and practiced in class previously. problems. Cognitive Level: Evaluate: Students will justify how they used proportions to solve

14 Sautter 14 Common Core State Standard: 7.G.1 Solve problems involving scale drawings of geometric figures, including computing actual lengths and areas from a scale drawing and reproducing a scale drawing at a different scale. Prior to this unit on proportions students will have set up proportions just not in the way we will be doing it. Previously they would have used proportions to set two numbers equal to each other so they could cross multiply to find a missing value. The difference now is that they will be using units in their proportions. Essentially the proportions will be set up the same way however students will need to remember that they must keep given information vertical and the same units horizontal. If they do not use this format then their proportion will not be set up correctly and they will solve to find a wrong answer. For this particular blueprint assessment point I have decided to use oral questioning. I plan on giving students a question on the board that will read: My car is 16 feet long and 8 feet wide. If I make it into a model that is 4 inches long, how wide will my model be? I will give them a few minutes to set up a proportion to solve this problem. I will then ask for a volunteer to write their proportion on the board. I will then ask students in the class to justify why certain information was placed where it was. Through this questioning I hope students will realize that the units of each ratio had to be identical. For example if the first ratio was set up using model in the numerator and actual in the denominator then that is how the second ratio must be constructed as well. Then going across the top you would have the model measurements and along the bottom would actual measurements. Additionally all given information would create one of the ratios. This would represent a valid proportion.

15 Sautter 15 I chose to use oral questioning for this assessment because it is something that I feel students need to see visually. If we discuss this topic as a class I will be able to highlight the units horizontally and the given information vertically to show that when setting up a proportion we always need this t as none of the information can be similar diagonally. It is my feeling that seeing this visually will help remind students to double check the setup of their proportions prior to solving them. Content Area: Applying Percent Cognitive Level: Understand: Students will explain their thought processes when finding percentages with or without a proportion. Common Core State Standard: 7. RP.2 Recognize and represent proportional relationships between quantities. C. Represent proportional relationships by equations. Throughout the course of this unit students will discover multiple ways of solving percent problems. One way they will recognize immediately is the proportion method since that will be what we will primarily have been working with prior to this lesson. However another way students could solve a percent problem is with the use of the percent equation. Both methods begin by writing a percent statement which would follow the form % of is. This will assist students as they create their proportion is/of = %/100. However another way to translate the percent statement is to take it literally word for word to create the percent equation: percent as a decimal times of equals is. Then students just need to plug and chug to find their missing number. To assess this blueprint point I have decided to use an exit slip (see Appendix D). At the end of the class period where we discussed this new concept of percent equation, I will ask

16 Sautter 16 students to describe their thought process in creating the percent equation as this equation does in fact represent a proportional relationship. Students will be expected to turn in a thoughtful response with examples prior to leaving class for the day. It is expected they will summarize the main points of the lesson from that day as we worked through creating this percent equation. I chose an exit slip as the formative assessment for this blueprint point because I think it is a good point for students to reflect on what they have learned about proportions thus far and how they relate to percentages. Additionally, it will lend me valuable information as to where my students are in understanding this concept and if they have any major misconceptions that need to immediately be addressed the following day. If students are able to provide a clear justification for their thought process and provide valid examples then they obviously have a clear understanding of the concept and the ability to represent a proportional relationship as an equation as the standards suggest. Cognitive Level: Apply: Students will demonstrate how they think about using percents in real life in relation to tax, tip, discounts, interest, etc. Common Core State Standard: 7.RP.3 Use proportional relationships to solve multistep ratio and percent problems. Connecting the mathematics content we cover in class to the real world is a major part of every math teacher s job. Every day student s ask when will I ever use this in the real world? As their teacher it is our obligation to provide students with multiple avenues of where this stuff we are teaching them will ever be used or seen again. The great thing about applying percents is that it is easily seen all over the real world and most students have seen or dealt with these concepts as they went out to dinner with their family or while they shopped at the mall with

17 Sautter 17 friends. Although many of them have heard of tax, tip, discounts, and etc., very few of them know how to calculate such percentages. This particular content area of the unit will be interesting for students to learn as they will be able to apply their knowledge of proportions to these real world concepts. Since many of my students will know about tax, discounts, and tips I will use oral questioning to assess their knowledge of these concepts. I will begin by asking students to create a list of ways they have seen or know of percents being used in the real world. It is here where I expect many of my students to think of the concepts such as tax, tip, discount, interest, and etc. I will then ask them to think about how these concepts can be applied to what we have learned in class. I expect a few of my students to come up with the idea that we could use proportions to solve for these percentage values. Before we begin to solve these types of problems we will first need to define these words as each one has a very special meaning when we apply them to percent problems. For example if we are calculating tax we need to know that this is an additional amount on the price of the original item. Whereas, if we are calculating a discount, students will need to know that we will need to subtract this discount percentage from the original amount of the item. Therefore vocabulary will be a very vital part of this content area. I decided to use oral questioning to assess this blueprint point since this is not something new to my students. Each and every one of them has come across percent problems in their everyday life outside of school at one point or another. Therefore since it is common knowledge I believe we can have a very good conversation about the topic and how it relates to proportions.

18 Sautter 18 Cognitive Level: Create: Students will create their own situations to determine items such as tax, tip, discounts, interest, commission, etc. Common Core State Standard: 7.RP.3 Use proportional relationships to solve multistep ratio and percent problems. I chose this assessment point because I feel that if students are not only able to solve but create their own multi-step and percent problems to solve then they must have a strong understanding of the concept. It is one thing if students can solve a multistep ratio or percent problem but if they can create one themselves, then that shows a higher level of critical thinking that I want all my students to strive for. To assess this blueprint point I decided to use a free write. I would give students time to create their own multistep ratio or percent problem. For this they would be expected to create a little word problem that would require one of the methods learned in class to solve. As they write these problems I will circulate the room reading various questions students create. After everyone is finished I will have them trade papers with their learning partners. It is now when I will have students actually try to solve their learning partner s problem. Before the end of class I will collect everyone s paper to see the question they created and the solution their learning partner gave. By reading the types of problems and solutions students created I will be able to assess their ability to write and solve a multi-step or percent problem. I chose to do a free write for this assessment point because I wanted to give my students the opportunity to be creative and really show what they have learned about ratios and percents. This free write will take place towards the end of the unit so students will have a wide array of content to pick from as they create this problem for their learning partner to solve.

19 Sautter 19 Reflection As I began to create this blueprint I found it very difficult to try to put point values on each point. I knew that my point values needed to align with the Common Core State Standards as well as my personal feelings on the importance of each point however it was still difficult to say this point is only worth two percent while this point should be worth twelve. I think I had a difficult time doing this because I knew what the standards wanted me to cover but I had to determine what would be beneficial for my students trying to meet these standards. In the end I feel as though I successfully assigned point values as each content area resulted in similar percentages of the entire unit, just as I had hoped for. Before beginning to assign points I felt that each content area deserved equal weight in the test blueprint as each area was as important as the next. Overall, I feel that my test blueprint shows a fair representation of the content in my ratio and proportions unit in relation to Bloom s taxonomy. Reflecting on the learning targets I have set for my students, I feel that my use of assessments across Bloom s levels of understandings provides my diverse class with diverse assessments. I feel very confident that my ability to assess across all levels, particularly more so at the higher levels such as analyze and evaluate, lends great benefits to my diverse group of students. Throughout the unit I provide students with a wide variety of assessments to show what they have learned about the content thus far. I will not solely be assessing my students on one test or quiz where the reliability and validity would be low as social and health factors may reduce the validity of the scores on that one day. Although I do have one quiz as an assessment, it may be beneficial for me to give the quiz to the students a few days later to test the validity of the results. If the students receive similar or higher scores the second time around then that would prove my assessment is valid. However if students receive

20 Sautter 20 lower scores the second time then that will show me I need to rethink my assessment as this is not showing me valid, reliable results. Another way I may test the reliability and validity of my assessments mentioned in this paper would be to have another colleague sit in on one of my classes to give me an outsiders view on the validity of my formative assessments. One way to test the validity of my free write assessment would be to show it to another math teacher and have them grade the student on their problem writing skills and procedure solving. If we both agree on the score of the writing and solving then that would also prove my assessment to be valid. With all of the formative assessments discussed I would expect generally valid, thought out responses from my students. Much of this unit has a real-world application and I feel as though this will greatly benefit my students interest in the material. With any and all of the assessments much of the feedback I will provide students with whether written or oral, will be in the form of a question. It is my belief that by asking them a question such as why did you do this or how did you get there I can find their misconception and hopefully lead them into the correct solution without coming out and explicitly telling them you need to do this. If they do not know where they went wrong and why that was wrong they are bound to do it again on the next problem. As their teacher I want to help them correct their mistakes early on to avoid unnecessary frustrations later. If I see many of my students having the same misconceptions on a particular concept then I feel it is my responsibility to reflect on my lesson and look to find another way of explaining the material the following day. Because if the majority of my class is having difficulty then my method of teaching them the concept must not have been clear and it is my job to present the material in a clear way to them. Therefore the amount and type of instructional adjustment

21 Sautter 21 needed after any one of these formative assessments will be dependent on the number of students needing additional assistance and the format in which this group of students learns best whether it be visually or audial. As a teacher, I now see how creating a test blueprint can be helpful as we try to formulate valid, reliable assessments to measure our students knowledge throughout a unit. It is very easy to get caught up in the curriculum forgetting about our long term goals.the standards.

22 Sautter 22 Appendix A KWL Chart Name: Date: Period: K What I KNOW about Ratios Where have you seen ratios used? W What I WANT to know about Ratios L What I LEARNED about Ratios What is a ratio? An example of a ratio is

23 Sautter 23 Appendix B Quiz Name: Date: Period: Directions: Answer all questions using complete sentences and be sure to include a unit rate when applicable. 1. Determine whether the set of numbers in the table is proportional. Explain. Quiz Number of Socks Cost $2 $4 $6 $8 2. The amount of time it takes to cook a turkey increases with the weight of the turkey. It is recommended that you cook a 10-lb turkey for 3 hours. An extra 12 minutes of cooking time is necessary for each additional pound of turkey. Is the cooking time proportional to the weight of the turkey? Explain your reasoning. 3. The equation of a line is y=6x. Is this line proportional? Why or why not?

24 Number of Packs Sautter Fruit Snacks Number of boxes Is this graph proportional? Why or why not?

25 Sautter 25 Appendix C Homework Name: Date: Period: Directions: For each graph or table state whether or not the relationship is proportional. Then in one sentence describe the relationship being shown and state what one point (x,y) represents.

26 Sautter 26 Appendix D Exit Slip Name: Date: Period: Describe your thought process in creating the percent equation. Be sure to include valid examples.