DEPARTMENT OF THE AIR FORCE Thomas N. Barnes Center for Enlisted Education (AETC) Maxwell AFB, AL 36118 ENLISTED PME INSTRUCTOR COURSE STUDENT READING 1 Sep 14 LESSON TITLE: TF08, ASSESSING, EVALUATING, AND OBSERVING (AEO) REFERENCES: Angelo, T, and K.P. Cross. Classroom assessment techniques a handbook for college teachers. SanFrancisco: Jossey-Bass A Wiley Imprint, 1993. "Assessing and Evaluating Student Learning." Atlantic Canada Englis Language Arts Curriculum, 2010: 263-278. Assessment - Center for Teaching and Learning. 2013. https://ctl.utexas.edu/teaching/assessment. Barton, Linda G. Quick Flip Questions for Critical Thinking. Madison, Wicsonsin: EDUPress, 2013. Bloom, Benjamin S. Taxonomy of Educational Objectives. New York City, New York: Longman Inc., 1956. Definitions of Bloom's Taxonomy. 2010. http://enpub.fulton.asu.edu/mcneill/blooms.htm. Haynes, Marion E. Managing Performance: A Comprehensive Guide to Effective Supervision. Los Altos, California: Crisp Publications Inc., 1984. Methods of Assessment - Center for Teaching and Learning. 2013. https://ctl.utexas.edu/assessment/methods. Taylor-Powell, Ellen, and Sara Steele. "Collecting Evaluation Data: Direct Observation." University of Wisconsin: Program Development and Evaluation, 2010. Using Rubrics. 2013. http://www.cte.cornell.edu/teaching-ideas/assessing-studentlearning/using-rubrics.html. What is peer observation? - Center for Teaching and Learning. 2013. http://ctl.utexas.edu/node/245. TERMINAL COGNITIVE OBJECTIVE: Comprehend Assessing, Evaluating, and Observing concepts and their impact on educator, student, and mission effectiveness. TERMINAL COGNITIVE SAMPLES OF BEHAVIORS: 1. Explain Assessing, Evaluating, and Observing concepts and their relationship to educator, student, and mission effectiveness. 2. Give examples of Assessing, Evaluating, and Observing concepts and their impact on educator, student, and mission effectiveness.
3. Predict the impact of Assessing, Evaluating, and Observing concepts on educator, student, and mission effectiveness. Assessment Assessing and Evaluating Assessment focuses on learning, teaching and outcomes. It provides information for improving learning and teaching. Assessment is an interactive process between students and faculty that informs faculty how well their students are learning what is taught. This information obtained through assessment allows staff and faculty to make changes in the learning environment, and is typically shared with students to assist them in improving their learning and study habits. This information is learner-centered, course based, frequently anonymous, and not graded. One method used for assessing in EPME is a formative assessment. Formative assessment techniques monitor student learning during the learning process. The feedback gathered is used to identify areas where students may be struggling so instructors can adjust their teaching and students can adjust their studying/learning. These are low-stakes assessments. It is important to understand that formative assessments are not just tests; we do these all the time. Some examples of assessments used in academia are: Written reflections: Sometimes referred to as "Minute Papers" or "Muddiest Points," these popular assessment techniques have students reflect immediately following a learning opportunity (e.g., at the end of a class or after completing an out-of-class activity) to answer one or two basic questions like: - What was the most important thing you learned today? - What was the most confusing topic today? - What important question remains unanswered? Polls/Surveys: Data on student opinions, attitudes, behaviors or confidence in understanding can be gathered either during class (e.g., with a classroom response system) or outside of class (e.g., during feedback sessions with students). This can illustrate student engagement with the material as well as prior knowledge, misconceptions, and comprehension. Checks for Understanding: Pausing every few minutes to see whether students are following along with the lesson not only identifies gaps in comprehension, but helps break up lectures (e.g, with random questions to see if students are following or online lessons, or with embedded quiz questions) into more digestible bites. Lesson wrappers: These include questions at the beginning of class about what students anticipate getting out of a lesson and/or questions at the end of class about the key points of the lesson. Having students compare their key points to the instructors can help students develop skills in active listening and identifying important information TF08SR - 2
In-class activities: Having students work in pairs or small groups to solve problems creates space for powerful peer-to-peer learning and rich class discussion. Instructors should roam the classroom as students work, helping those who may be struggling and guiding those who are headed in the wrong direction. Quizzes: Gauge students prior knowledge, assess progress midway through a unit, create friendly in-class competition, review before the test -- quizzes can be great tools that don't have to count heavily toward the students' grades. Using quizzes to begin units is also a fun way to assess what your students already know, clear up misconceptions, and drive home the point of how much they will learn. Evaluation Evaluation is a process that focuses on grades or scores which reflect mastery level of all course content. These could include discussion, cooperation, attendance, and verbal ability. One method used for evaluation in EPME is the summative evaluation. Summative evaluations evaluate student learning. These are high-stake evaluations that typically occur at the end of an instructional unit or course and measure the extent to which students have achieved the desired learning outcomes. Some examples of evaluations used in academia are: Exams: This includes mid-term exams, final exams, and tests at the end of course units. The best tests include several types of questions short answer, multiple-choice, true-false, and short essay to allow students to fully demonstrate what they know. Papers, projects, and presentations: These give students the chance to go deeper with the material to put the knowledge they ve acquired to use or create something new from it. This level of application is an extremely important and often overlooked part of the learning process. These types of projects also give students who do not test well a chance to shine. The table shown here illustrates the differences between assessment and evaluation Dimension of Difference Assessment Evaluation Content: timing, primary purpose Orientation: focus of measurement Findings: uses thereof Formative: ongoing, to improve learning Process-oriented: how learning is going Diagnostic: identify areas for improvement Figure 1. Assessment vs. Evaluation Summative: final, to gauge quality Product-oriented: what s been learned Judgmental: arrive at an overall grade/score TF08SR - 3
Observing Observing and the BCEE IEC Observation is the skill of using as many of the senses as possible in an effort to obtain information; typically watching and listening. There are two ways to observe; indirect and direct. Indirect observation is learning information from another source, such as a book or another person. During indirect observation the individual is not present and the observation is done after the fact. There are some instances in EPME where indirect observation is utilized. Some instances include counseling, discussion with other instructors, and end of course surveys. Direct observation is using as many of the senses as possible in an effort to obtain information; typically watching and listening. There are a multitude of times in EPME when you will use direct observation. Some of these are evaluating speeches, leadership labs and experientials. Since a good portion of your time in EPME will be using direct observation, the next portion of this section is devoted to direct observation. The table shown here illustrates what to look for during direct observation of a student Observing Individual characteristics What to look for Attitude towards subject Desire to learn Wanting to be there Interest and commitment Interactions with group Level of participation in group Power relationships in group (leaders, followers) Climate towards learning Level of cooperation Ability to problem solve Nonverbal behavior Facial expressions (communicate understanding or confusion) Gestures Posture TF08SR - 4
Eye contact Note taking Beverage intake (trying to stay awake with energy drinks) When presenting Clarity of communication Ability to answer questions Knowledge of subject Use of instructional aids Understanding of assignment Figure 2. Direct Observation When using observation to collect information, you need to watch and listen attentively. There are four elements that can help you to hone your direct observation skills. These are capturing detail, discerning, interpreting, and validating: Capturing detail: This involves looking at all the details of what is being observed and being able to focus on more than one aspect of what is being observed. Discerning: There is likely to be much more in what you are observing than you can or need to take in. You need to discern what is important to observe by knowing what your goal in observation is. You must be able to filter out anything that does not support your goal. Interpreting: You must be able to take what you have discerned and interpret that information or data in accordance with the goal of your observation. While interpreting, observers often incorrectly associate their values and beliefs to what they observe. Validating: Often, a decision must be made about what is observed. If that is the case it is important to validate that decision. This can be done by repeating the observation (if feasible), asking others who also observed, or speaking with someone who is knowledgeable in the subject area of what is observed. The last type of observation we use in EPME is peer observation. Peer observation can be divided into two sections: student to student observation and instructor to instructor observation. Student to student observation This portion of peer observation provides an opportunity for students to discuss performance one with another. This can either be a formal or informal process where students talk with one another about strengths and weaknesses. Students can also talk one TF08SR - 5
with another about what they have learned about course. Some of the best learning in an EPME course is often accomplished via informal student peer observation. This type of peer observation can produce the following: - Comments on the relationship between instructor acts and student behaviors - Comparison with methods peers consider to be good Instructor peer observation This portion of peer observation involves instructor peers who review an instructor s performance through classroom observation and examination of instructional materials including the course design. Observations of classroom behavior are intended for reviewing the teaching process and its possible relationship to learning. The focus is on verbal and nonverbal behaviors of both the instructor and the students in the classroom. This type of peer observation can produce the following: - Specific suggestions for instructors to improve teaching - Dialogues about teaching strategies and best practices Instructor to instructor observation in EPME typically occurs in the form of evaluation. Thus, a more appropriate term (though not used in academia) would be evaluator to instructor observation. In EPME it is very likely the individual evaluating you will be your peer (same rank, duty title, etc.) The tool used in EPME for this type of observation is the EPME 620 Barnes Center for Enlisted Education (BCEE) Instructor Evaluation Checklist (IEC) According to the Barnes Center Instruction 36-2301, USAF BCEE Procedural Guidance a strong faculty evaluation program is the most effective way to improve teaching skills, to ensure instructors teach the curriculum as designed, and to identify training requirements and deficiencies. One tool in doing this is the BCEE IEC. Solid faculty evaluation programs go beyond minimum evaluation requirements and include all of the following components: - Thorough and purposeful written feedback directed at improving instructor effectiveness or lesson deliver. Focus feedback comments on instructional strengths, areas for improvement, and action plans for achieving necessary improvements. - Semiannual reviews of all faculty evaluations occurring during that period to determine if opportunities exist for improving instructional delivery capabilities. - Use of summary and test item statistics to determine if instructors require additional evaluations. - Additional instructor evaluations if an ARB discovers an institutional failure to conduct the instructional program as designed. - Evidence that demonstrates use of the EPME Performance Indicators. TF08SR - 6
Using Rubrics A rubric is a guide listing specific criteria for grading or scoring academic papers, projects or tests. A rubric is a type of scoring guide that assesses and articulates specific components and expectations for an assignment. Rubrics can be used for a variety of assignments: research papers, group projects, portfolios and presentations. Rubrics help instructors: - Assess assignments consistently from student-to-student. - Give timely, effective feedback and promote student learning in a sustainable way. - Clarify expectations and components of an assignment - Refine teaching skills by evaluating rubric results. - Save time in grading, both short-term and long-term. Rubrics help students: - Understand expectations and components of an assignment. - Become more aware of their learning process and progress. - Improve work through timely and detailed feedback. Rubric development guidelines - Outline the elements or critical attributes to be evaluated (these attributes must be objectively measurable). - Create an evaluative range for performance quality under each element; for instance, excellent, good, unsatisfactory. - Add descriptors that qualify each level of performance. - Avoid using subjective or vague criteria such as interesting or creative ; instead, outline objective indicators that would fall under these categories. - The criteria must clearly differentiate one performance level from another. - Assign a numerical scale to each level. - Train students to use your rubric and solicit feedback; this will help you judge whether the rubric is clear to them and will identify any weaknesses. - Rework the rubric based on the feedback. Biases and AEO You will experience varying pressures when assessing, evaluating and observing. According to Marion Haynes, author of Managing Performance: A Comprehensive Guide to Effective Supervision, evaluation errors can occur in the following areas: TF08SR - 7
Standards rating error bias: This is a tendency to over or underrate (high or low) compared with the average of other raters. Halo/Horns Effect: This error is occurs when one factor, either positive or negative, influences the evaluation of other factors. Central Tendency: This pitfall occurs when evaluators rate everyone as average. Personal Bias: This error occurs when the evaluator allows his or her personal likes and dislikes of the individual being rated to influence the evaluation. Logic rating error: This error is the tendency to give similar ratings to two or more traits logically related in the mind of the rater. Levels of Learning A primary reason for assessing, evaluating, and observing in EPME is to determine what level of learning students have reached. Bloom's taxonomy is a classification of learning objectives within education. It is named for Benjamin Bloom, who chaired the committee of educators which devised the taxonomy and edited the first volume of the standard text, Taxonomy of Educational Objectives: The Classification of Educational Goals. Bloom's taxonomy refers to a classification of the different learning objectives that educators set for students. It divides educational objectives into three domains: cognitive, affective, and psychomotor. For the purpose of this lesson, we will focus on the cognitive domain. Skills in the cognitive domain revolve around knowledge, comprehension, and critical thinking on a particular topic. Traditional education tends to emphasize the skills in this domain, particularly the lower-order objectives. There are six levels in the taxonomy, shown here lowest to the highest: TF08SR - 8
Figure 3. Blooms Taxonomy TF08SR - 9
Knowledge Being able to exhibit memory of previously learned material by recalling facts, terms, basic, concepts and answers. Key words associated with knowledge level: choose how match recall spell define label name relate tell find list omit show what Comprehension One s ability to demonstrate understanding of facts and ideas by organizing, comparing, translating, giving descriptions, and stating main ideas. This is a level of learning where the student can work problems associated with subject and explain what was done. Key words associated with comprehension level: classify demonstrate infer outline show compare explain illustrate relate summarize contrast extend interpret rephrase translate Application One s ability to solve problems to new situations by applying acquired facts, knowledge, techniques and rules in a different way. Key words associated with application level: build develop make use of plan choose experiment with model solve construct identify organize utilize Analysis One s ability to examine and break information into parts by identifying motives or causes. Also being able to make inferences and find evidence to support generalizations. Key words associated with analysis level: assumption contrast examine motive take part in categorize discover divide relationships test for classify dissect inference simplify theme TF08SR - 10
Synthesis One s ability to compile information together in different ways by combining elements in a new pattern or proposing alternative solutions. Students will be able to generate ideas and use them to create a physical object, a process, a design method, a written or oral communication, or even a set of abstract relations Key words associated with synthesis level: adapt compile improve originate solution build discuss invent predict test combine elaborate modify propose theorize Evaluation One s ability to present and defend opinions by making judgments about information, validity of ideas or quality of work based on a set of criteria A level of learning where the student can create a variety of ways to solve the problem and then, based on established criteria, select the solution method best suited for the problem Key words associated with evaluation level: appraise conclude justify prioritize recommend award deduct opinion prove rule on choose judge perceive rate support TF08SR - 11