Faculty Unplugged? Empowering Faculty in Today s Assessment Culture Presented by: Julie Basler, PhD (Vice President of Academic Affairs-Platt College) Juanita L. Gurubatham, PhD (Director of Institutional Review and Development-ACCSC) ACCSC Faculty Development Workshop February 11, 2015 Arlington, Texas
Today s Agenda Defining the Loop: Defining measureable course objectives, program objectives, and how to assess it all both internally and externally Creating the Loop: Tying course objectives to program objectives Enforcing the Loop: Using valid, clear course activities to achieve objectives Closing the Loop: Reaching program outcomes at the end of a program
Defining the Loop
Breaking it Down in Terms Course objective: Statements of expectation written in measurable terms that express what a student will know, do, or think at the end of a learning experience. Course objectives are measureable learneroriented abilities that are consistent with standards of a professional practice.
Breaking it Down in Terms Program Objectives: Two types: Internal Measurements and External Measurements Internal Measurements: Performance indicators that reflect the extent to which the purposes of the program are achieved and by which program effectiveness is documented. External Measurements: Consumer-Oriented indexes designed to evaluate the degree to which the program is achieving its mission and goals.
Breaking it Down in Terms Competencies: Measurable behaviors, knowledge, actions, and skills essential to the practice of a profession. Institutional Effectiveness: Written document based on measurable standards that reflect the process of ongoing comprehensive assessment of all program components, including program objectives, etc.
Taking it Back Home A fun and meaningful way to teach all of these terms is to use a map. A map provides major interstates as the Program Outcomes By-passes and highways as the competencies and so on Choose a map of the area of each person is from and let that person explain using his/her own home town area
How Does this Tie Together? Course Objective Competency Program Outcome(s) Faculty the key to this involvement How is faculty the key and the starting point to this whole process?
Why are Course Objectives Students can see how material is related to their educational goals Important? Exams correspond to the stated learning objectives (Once faculty have written learning objectives, they have defined their assessment materials.) Students know what they are expected to be able to do after the instruction is complete daily, quarterly, etc.
Why are Course Objectives Important? Course is organized (With objectives, the topics fit together and have direction.) In short, course objectives communicate what the instructor is trying to teach; what the students are to be expected to be able to do; how their achievement will be measured; and what will be accepted as evidence that they have achieved the goals.
Creating a Course Objective Step One: Determine where the course lies in the curriculum. Step Two: Determine what knowledge or skills your students have from previous courses. Step Three: Establish or Determine a Course Goal Step Four: Break down the course goal into subcomponents. Ask yourself what the specific knowledge, skills, and attitudes you expect the students to have at the end of the course, based on the course goal and where the course appears in the curriculum.
Creating a Course Objective Step Five: Rewrite these subcomponents into measurable units (aka learning objectives). These measurable units should contain the following criteria: A = Audience. Who will be completing the subcomponent? In the case of your class, it is always the student, therefore most people leave audience out of the learning objective. B = Behavior. What knowledge, skill or attitude do you expect the student to have? You can use Bloom s taxonomy as a guide to determine the level of behavior. C = Condition. Any special circumstances needed to complete the objective.
Example Course Objective Course goal: Students will be able to correctly identify anatomical structures in the human body. Specific learning objectives: At the completion of the course, students will be able to: Recall the attachments of the muscles of the arm. Identify the muscles of the arm on a model, picture, diagram, drawing, or specimen.
Example Course Objective Course goal: Students will learn common theories of learning and apply them to teaching preschool students. Specific learning objectives: At the completion of the course, students will be able to: Compare and contrast the learning theories of Vygotsky and Piaget. Design an appropriate playtime activity for three year olds using Piaget s theory From the VCU Center for Teaching Excellence
Taking it Back Home Don t ask for people to write course objectives on the spot: It causes faculty to freak out Instead, have them evaluate ones that you pull and them have them submit revised ones privately, not publicly Think of ways to praise faculty for well written course objectives: free things exist too: Car wash tokens Lottery scratch offs
Using Course Objectives Using Bloom s Taxonomy- Student Learning Outcomes A committee of colleges, led by Benjamin Bloom (1956), identified three domains of educational activities: Cognitive: mental skills (Knowledge) Psychomotor: manual or physical skills (Skills) Affective: growth in feelings or emotional areas (Attitude)
Categories of Bloom s Writing Student Learning Outcomes Using the Domains 1. Remembering 2. Understanding 3. Applying 4. Analyzing 5. Evaluating 6. Creating Taxonomy
Taking It Back Home Have Faculty Members Draw an Illustration in Stick People for Each of These! 1. Remembering 2. Understanding 3. Applying 4. Analyzing 5. Evaluating 6. Creating
Categories of Bloom s Taxonomy Remembering: Recall previous learned information. Examples: Recite the safety rules in administering insulin. Knows the steps of wrapping a sprained ankle. Key Words: defines, describes, identifies, knows, labels, lists, matches, names, outlines, recalls, recognizes, reproduces, selects, states.
Categories of Bloom s Taxonomy Understanding: Comprehending the meaning, translation, and interpretation of instructions and problems. State a problem in one's own words. Examples: Explain in one's own words the steps for creating an excel spreadsheet. Key Words: comprehends, converts, defends, distinguishes, estimates, explains, extends, generalizes, gives an example, infers, interprets, paraphrases, predicts, rewrites, summarizes, translates.
Categories of Bloom s Taxonomy Applying: Use a concept in a new situation or unprompted use of an abstraction. Applies what was learned in the classroom into novel situations in the work place. Examples: Apply laws of statistics to evaluate the reliability of a written test. Key Words: applies, changes, computes, constructs, demonstrates, discovers, manipulates, modifies, operates, predicts, prepares, produces, relates, shows, solves, uses.
Categories of Bloom s Taxonomy Analyzing: Separates material or concepts into component parts so that its organizational structure may be understood. Distinguishes between facts and inferences. Examples: Troubleshoot a piece of equipment by using logical deduction. Key Words: analyzes, breaks down, compares, contrasts, diagrams, deconstructs, differentiates, discriminates, distinguishes, identifies, illustrates, infers, outlines, relates, selects, separates.
Categories of Bloom s Taxonomy Evaluating: Make judgments about the value of ideas or materials. Examples: Select the most effective solution. Explain and justify a new budget. Key Words: appraises, compares, concludes, contrasts, criticizes, critiques, defends, describes, discriminates, evaluates, explains, interprets, justifies, relates, summarizes, supports.
Categories of Bloom s Taxonomy Creating: Builds a structure or pattern from diverse elements. Put parts together to form a whole, with emphasis on creating a new meaning or structure. Examples: Write a company operations manual. Integrates training from several sources to solve a problem. Key Words: categorizes, combines, compiles, composes, creates, devises, designs, explains, generates, modifies, organizes, plans, rearranges, reconstructs, relates, reorganizes, revises, rewrites, summarizes, tells, writes.
What are Competencies? Measureable behaviors, knowledge, actions, and skills essential to the practice of any profession. Competency Types: Knowledge Competencies - practical or theoretical understanding of subjects. Skill and Ability Competencies - natural or learned capacities to perform acts. Behavioral Competencies - patterns of action or conduct.
Sample Competencies Achievement of the Program Outcomes for the Bachelor of Science in Nursing will be measured using the following five competencies based on the national standards of the Essentials of Baccalaureate Education and the Prelicensure knowledge, skills, and attitudes delineated by QSEN: (1) Nursing Process, (2) Nursing Informatics, (3) Evidence-Based Practice
Lunch More importantly -- Dessert
Creating the Loop
Course Objective To Program Objectives What do I want my graduate to look like when s/he graduates? What broad headers do I want them to fall under? Internal vs External-What s the Difference?
Taking It Back Home What do I want my graduate to look like when s/he graduates? Have people draw this out Crayons, paper, etc.
Program Outcomes The end result of what the graduate looks like at the end of the entire program on graduation day
Program Outcomes What the graduate looks like-internal Also, the program indices play a part in the external measurement (i.e. pass rates, placement rates, etc.)
Program Outcomes A program outcome must flow directly from, and support, the school and program/department mission. The connection between the mission and the outcome should be clear. A program outcome must be directly related to the discipline of the program. Consider these examples: General: Graduates of the Criminal Justice program will be critical thinkers. Program-specific: Graduates of the Criminal Justice program will analyze a current issue in criminal justice, evaluate evidence, and construct an argument.
Enforcing the Loop
ASSESSMENT FOR LEARNING ASSESSMENT OF LEARNING USING CLASSROOM ASSESSMENT TO ENGAGE STUDENTS
Assessment FOR Learning Turns the classroom assessment process and results into an instructional intervention Students and instructors work as a team Provide students with a clear vision of the learning target Provide students with descriptive feedback-show them how to improve. Accuracy (of Information) + descriptive feedback + student involvement = achievement gains Stiggins, R. Assessment FOR Learning Defined and Classroom Assessment for Student Learning
7 Strategies of Assessment For Learning Where am I going? 1. Provide a clear and understandable vision of the learning target. 2. Use examples of models of strong and weak work. Where am I now? 3. Offer regular descriptive feedback. 4. Teach students to self-assess and set goals. How can I close the gap? 5. Design lessons to focus on one aspect of quality at a time. 6. Teach students focused revision. 7. Engage students in self-reflection, and let them keep track of and share their learning.
Three Basic Questions for Evaluating Courses Are the learning objectives of the course being met? Are students being inspired and motivated to think analytically and creatively and to develop habits of mind appropriate to the discipline? Instruments of Evaluation Measures of student learning based on students in-course papers, projects, or exams evaluated by the faculty member s explicit standards and criteria Students performance on standardized tests Students performance in subsequent courses Are the course material, concepts, and activities rigorous, current, relevant for students needs, and consonant with the announced course description? Colleague examination of course syllabus, exams, and other material Do students perceive themselves to be well taught? Student course evaluations, surveys, focus groups Alumni surveys Walvoord, B.E., Assessment Clear and Simple
Taking it Back Home Using a MAP at the Course Level. Remind faculty that course objectives/outcomes should not be a secret. A map provides major interstates as the Course Objectives By-passes and highways as the competencies and so on Encourage faculty to set the destination in the first class period and continue to evaluate progress
Tools: Grades Exams and assignments must actually measure learning goals Establish criteria for assessment Develop systematic ways of to communicate student weaknesses and strengths to decision makers
Tools: Rubrics Allows for consist evaluation Provides various criteria and standards that may be used to evaluate student work May be used at course level and program level
Sample Rubric Trait 14: Sentence Construction Level 5 Clear and concise sentences vary, with the degree of complexity reflecting the audience and purpose. 4 Sentences vary, with the degree of complexity reflecting the audience and purpose. 3 Sentence variety is limited but attempts complex structure. 2 Complex structure is attempted without success and/or sentence structure is simplistic, but not throughout the text 1 Sentences are simple and repetitive. Cognitive Level and Quality of Writing Assessment : CLAQWA Online, Cross-Disciplinary. Retrieved from http://claqwa.com
Tools: E-Portfolio Portfolios contain authentic evidence of learning that students produce in the everyday course of their studies, both in and out of the classroom (e.g., in internships, independent research projects, and other real-world experiences). Enables students to collect and reflect on evidence of what they are learning. May be used at course level or program level Banta, T., et al, Three Promising Alternatives for Assessing College Students Knowledge and Skills
Tools: E-Portfolio These basic questions need to be answered when undertaking an eportfolio project. They include the following: When and from where will student work be collected? Are there specific assignments that correspond to specific outcomes? Is there a grading rubric? Who will evaluate the student eportfolios (if Programmatic Assessment)
Taking it Back Home Break faculty into groups by subject area Discuss/identify best tools for assessing competencies
Assessment OF Learning Assessments that happen after learning occurred to determine if it did. Primary users include policy makers, program planners, faculty, etc. Used to certify student competence
Tools: Standardized Exams Assessment OF Learning Useful in specific content areas Quantitative measure with established benchmarks Has limitations Licensure and Certification Exams
Program Assessment Now you have a program curriculum is it right? Assessment is an ongoing process Develop a plan Keep it simple Collect and evaluate your data Remember to close the loop Use results to improve curriculum
Taking it Back Home Course-level assessments of student learning should map to program assessment Program faculty should think collectively about how course assessments contribute to the program review. Keep it Simple Consider preset dates for annual program review Limit the number of objectives reviewed each year Put someone in charge
Building Blocks for Program Assessment Program Mission Statement of purpose and intentions Should connect to the institutional mission Provides foundation for development of goals and objectives
Building Blocks for Program Assessment Goals and Objectives What do you want a graduate to look like Must be measureable More is not always better
Building Blocks for Program Assessment Evaluation Determine what you will assess Evaluation tools Direct Assessment Indirect Assessment Who When Close the loop documentation!
Mission Goals/Outcomes Assessment Curricular Design Pedagogy AAC& U (2004), Taking Responsibility for the Quality of the Baccalaureate Degree
Taking it Back Home Planning your Day Incentive for participating Don t overwhelm participants Keep it simple