An Assessment Palooza: Engaging Faculty in the Assessment Process Presented by Susan C. Warner Taylor AAC&U 2018 General Education and Assessment February, 2018 Sponsored by:
Who We Are Baldwin Wallace University Baldwin Wallace University is an academic community committed to the liberal arts and sciences as the foundation for lifelong learning.
Who We Are Baldwin Wallace University Baldwin Wallace University is an academic community committed to the liberal arts and sciences as the foundation for lifelong learning.
University Undergraduate Learning Outcomes 1. Communication 2. Critical Thinking 3. Quantitative Literacy 4. Cultural Knowledge and Competence 5. Civic Engagement 6. Depth of knowledge in chosen discipline
COMMUNICATION WRITTEN COMMUNICATION Written communication is the development and expression of ideas in writing. Written communication involves learning to work in many genres and styles. It can involve working with many different writing technologies, and mixing texts, data, and images. Written communication abilities develop through iterative experiences across the curriculum. ORAL COMMUNICATION Oral communication is a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners attitudes, values, beliefs, or behaviors.
. AAC&U Value Rubric: Communication Capstone 4 Milestones 3 2 Benchmark 1 Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s). Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the assigned task(s) and focuses all elements of the work. Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assigned task(s) (i.e. the task aligns with audience, purpose, and context). Demonstrates awareness of context, audience, purpose, and to be assigned task(s) (i.e. begins to show awareness of audience s perceptions and assumptions). Demonstrates minimal attention to context, audience, purpose, and the assigned task(s) (i.e. expectation of instructor or self as audience). Content Development Uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer s understanding, and shaping the whole work. Uses appropriate, relevant, and compelling content to explore ideas within the context of the discipline and shape the whole work. Uses appropriate and relevant content to develop and explore ideas through most of the work. Uses appropriate and relevant content to develop simple ideas in some parts of the work. Genre &Disciplinary Conventions Formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields (please see glossary). Demonstrates detailed attention to and successful execution of a wide range of conventions particular to a specific discipline and/or writing task(s) including organization, content, presentation, formatting, and stylistic choices. Demonstrates consistent use of important conventions particular to a specific discipline and/or writing task(s), including organizations, content, presentation, and stylistic choices. Follows expectations appropriate to a specific discipline and/or writing task(s) for basic organizations, content, and presentation. Attempts to use a consistent system for basic organization and presentation. Sources & Evidence Demonstrates skillful use of highquality, credible, relevant sources to develop ideas that are appropriate for the discipline and genre of the writing. Demonstrates consistent use of credible, relevant sources to support ideas that are situated within the discipline and genre of the writing. Demonstrates an attempt to use credible and/or relevant sources to support ideas that are appropriate for the discipline and genre of writing. Demonstrates an attempt to use sources to support ideas in the writing. Control of Syntax and Mechanics Uses graceful language that skillfully communicates meaning to readers with clarity and fluency, and it virtually error- free. Uses straightforward language that generally conveys meaning to readers. The language in the portfolio has few errors. Uses language that generally conveys meaning to readers with clarity, although writing may include some errors. Uses language that sometimes impedes meaning because of errors in usage.
Context and Purpose BW Revised Rubric: Communication Superior Effective Emerging Ineffective N/A Score/Level Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the task and focuses on all elements of the work. Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the task. Demonstrates awareness of context, audience, purpose related to task. Demonstrates minimal attention to context, audience, and purpose. From Written Communication VALUE rubric (new to this draft) Content Development Uses appropriate, relevant, and compelling content to illustrate command and understanding of the subject and shape the whole work. Uses appropriate, relevant, and compelling content to explore ideas within the context of the subject and shape the work. Uses appropriate and relevant content to develop and explore ideas through most of the work. Uses some appropriate and relevant content to develop simple ideas in some parts of the work. From Written Communication VALUE rubric (part of Fall draft) Sources and Evidence Demonstrates skillful use of high-quality, credible, relevant sources to develop ideas in the work. Demonstrates consistent use of high-quality, credible, relevant sources to develop ideas in the work. Demonstrates an attempt to use credible and/or relevant sources to develop ideas in the work. Demonstrates an attempt to use sources to support ideas in the work. From Written Communication VALUE rubric (part of Fall draft) Use of Language Uses graceful language appropriate to the context that skillfully communicates meaning with clarity and fluency. Uses straightforward language appropriate to the context that consistently conveys meaning with clarity. Uses language that is usually appropriate to the context and generally conveys meaning with clarity. Uses language that lacks clarity and may not be appropriate to the context. From Written Communication VALUE rubric (leaves out criteria on mechanics) [Revision of Fall draft) Use of Media Uses media in highly effective way, enhancing communication and understanding of ideas. Uses media to contribute to effective communication of ideas. Uses media in way that is partially effective in communicating ideas. Uses media in way that does not effectively communicate ideas. Part of Fall draft
Overview of course-embedded assessment Faculty craft their own assignments for their courses A random sample of core courses that emphasize the ULO for that year Assignments specifically designed to address the ULO of interest will be uploaded to Blackboard and then entered into the assessment artifact database. (All assignments are rendered anonymous for assessment purposes.) Collect student artifacts in FYE, writing extension courses and capstone courses Ask instructors for the writing assignment These assignment files are then transferred into Aqua for our year-end Assessment Palooza.
Preparing for a Palooza Setting up project in Aqua (Following slides include screenshots from Aqua)
Assignmen Submissio t Name n ID Student Name Student ID Course Code Course Name Section ID Section Name Term Name Scoring Status Learning Outcome Criterion Faculty Intention Score 1 Evaluator Name - Score 1 Evaluator Enrollment - Score 1 Benjamin Course 1 85067Feuer 547850 Lies. Damned Lies & Statistics S16 S16-1 4Maria Mathews Benjamin Course 1 85067Feuer 547850 Lies. Damned Lies & Statistics Written Context Communic and S16-1 Fall 2016 Completed ation Purpose S16 S16-1 3Maria Mathews Benjamin Course 1 85067Feuer 547850 Lies. Damned Lies & Statistics Written Content Communic Developm S16-1 Fall 2016 Completed ation ent S16 S16-1 3Maria Mathews Benjamin Course 1 85067Feuer 547850 Lies. Damned Lies & Statistics Written Sources Communic and S16-1 Fall 2016 Completed ation Evidence S16 S16-1 4Maria Mathews Charles Course 1 85068Ray 543277 Lies. Damned Lies & Statistics Written Communic Use of S16-1 Fall 2016 Completed ation Language S16 S16-1 2Leslie Hunter Charles Course 1 85068Ray 543277 Lies. Damned Lies & Statistics Written Context Communic and S16-1 Fall 2016 Completed ation Purpose S16 S16-1 2Leslie Hunter Charles Course 1 85068Ray 543277 Lies. Damned Lies & Statistics Written Content Communic Developm S16-1 Fall 2016 Completed ation ent S16 S16-1 2Leslie Hunter Charles Course 1 85068Ray 543277 Lies. Damned Lies & Statistics Written Sources Communic and S16-1 Fall 2016 Completed ation Evidence S16 S16-1 2Leslie Hunter Christina Course 1 85069Collins 545961 Lies. Damned Lies & Statistics Written Communic Use of S16-1 Fall 2016 Completed ation Language S16 S16-1 Written Context Communic and S16-1 Fall 2016 Completed ation Purpose 2Judette Kullins
Assessment Results
Assessment Palooza Day 42 faculty working in pairs Start with calibration 12-16 papers Consensus on the scoring of the rubric
PA.. ZA! LOO.. PALOOZA CHAT
#lovemypalooza
Almost 4 pm and going strong
Using Aqua for Core Assessment
What did the results of the Assessment Palooza reveal? Quantitative Analysis of Effective Writing Data
Sample Demographics (N=242 papers) Student authors: 66% Female (160) 17% Students of Color (42) Class distribution: FY 28% (68) SO 18% (44) JR 18% (44) SR 36% (86) 15% First-generation college students (37)
Writing Outcomes Mean Ratings Analysis (All differences are statistically significant, p<.001)
Interesting patterns in the data (Keep in mind that all data are cross-sectional) Across the board, Seniors write better than first-year students. Gains in all 4 aspects of writing tend to be linear. Interestingly, there is a statistically-significant difference between JRs and SRs on all aspects of writing but Sources & Evidence. (There is no significant difference between JRs and SRs on that scale.) On average, female students are better writers than male students. Female students writing development tends to be linear. Male students initially show slow/no growth, then dramatic gains in JR and SR years. By senior year, writing skills of female and male students are comparable.
Writing Outcomes Ratings Distributions Context & Purpose Content Development
Writing Outcomes Ratings Distributions Sources & Evidence Use of Language
Results presented at Fall Conference Panel discussion with Palooza participants
How we are using the data Assessment committee prepared a report Evaluation by Core Committee Workshops offered by Center for Teaching Excellence
Questions?
Discussion Q&A How would you go about assessing the faculty assignments given to students for a written paper?
Up Next Critical Thinking Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion. Quantitative Literacy A Habit of mind, competency, and comfort in working with numerical data. Individuals with strong QL skills possess the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations. They understand and can create sophisticated arguments supported by quantitative evidence and they can clearly communicate those arguments in a variety of formats (using words, tables, graphs, mathematical equations, etc., as appropriate
AAC&U Value Rubric : Cultural Knowledge & Competence Capstone 4 Milestones 3 2 Benchmark 1 Knowledge Cultural self-awareness Articulates insights into own cultural rules and biases (i.e. seeking complexity; aware of how her/his experiences have shaped these rules, and how to recognize and respond to cultural biases, resulting in a shift in self- description). Recognizes new perspectives about own cultural rules and biases (i.e. not looking for sameness; comfortable with the complexities that new perspectives offer). Identifies own cultural rules and biases (i.e. with a strong preference for those rules shared with own cultural group and seeks the same in others). Shows minimal awareness of own cultural rules and biases (even those with own cultural group(s)) (i.e. uncomfortable with identifying possible cultural differences with others). Knowledge Knowledge of cultural worldview frameworks Demonstrates sophisticated understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Demonstrates adequate understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Demonstrates partial understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Demonstrates surface understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy, or beliefs and practices. Skills Empathy Interprets intercultural experience from the perspectives of own and more than one worldview and demonstrates the ability to act in a supportive manner that recognizes the feelings of another cultural group. Recognizes intellectual and emotional dimensions of more than one worldview and sometimes uses more than one worldview in interactions. Identifies components of other cultural perspectives but responds in all situations with own worldview. Views the experience of others but does so through own cultural worldview. Skills Verbal and nonverbal communication Articulates a complex understanding of cultural differences (i.e. demonstrates understanding of the degree to which people use physical contact while communicating in different cultures or use direct/indirect and explicit/implicit meanings) and is able to skillfully negotiate a shared understanding based on those differences. Recognizes and participates in cultural differences in verbal and nonverbal communication and begins to negotiate a shared understanding based on those differences. Identifies some cultural differences in verbal and nonverbal communication and is aware that misunderstandings can occur based on those differences but is still unable to negotiate a shared understanding. Has a minimal level of understanding cultural differences in verbal and nonverbal communication; is unable to negotiate a shared understanding. Attitudes Curiosity Asks complex questions about other cultures, seeks out and articulates answers to these questions that reflect multiple cultural perspectives. Asks deeper questions about other cultures and seeks out answers to these questions. Asks simple or surface questions about other cultures. States minimal interest in learning more about other cultures. Attitudes Openness Initiates and develops interactions with culturally different others. Suspends judgment in valuing her/his interactions with culturally different others. Begins to initiate and develop interactions with culturally different others. Begins to suspend judgment in valuing her/his interactions with culturally different others. Expresses openness to most, if not all, interactions with culturally different others. Has difficulty suspending and judgment in her/his interactions with culturally different others, and is aware of own judgment and expresses a willingness to change. Receptive to interacting with culturally different others. Has difficulty suspending any judgment in her/his interactions with culturally different others, but is unaware of own judgment.
Contact Us Susan Warner Taylor Director of Institutional Research & Assessment swarner@bw.edu Watermark events@watermarkinsights.com