Closing the Widest Loops: Reflecting on Outcomes across Stakeholder Groups Dr. Diana Ashe Dr. Matthew TenHuisen Dr. Andy Mauk University of North Carolina Wilmington December 3, 2017 University of North Carolina Wilmington One of 17 University of North Carolina institutions 16,500 total students 14,500 undergraduates Doctoral Granting, Regional Comprehensive Focused on Undergraduate Applied Learning and Teaching Excellence UNCW Fast Facts 55 majors 4 doctoral degrees 33 master s degrees Organized into 4 Colleges and Schools One of the highest 4 year graduation rates and the highest transfer graduation rate of all NC public universities More than 74% of incoming students surveyed indicate that UNCW was their first choice More than 13,000 first year students applied for one of 1,970 spaces in the most recent entering class 16.4% of FY students, 16% of transfer students, and 22 percent of faculty identify themselves as members of a minority population 1
Culture of Assessment @ UNCW Embedded Assessment Directors Academic Colleges Student Affairs Business Affairs Institutional Research & Planning Office (Institutional Effectiveness) Learning Assessment Council Manages Taskstream for Assessment Reporting SACSCOC Workgroup UNCW Learning Goals Institutional learning goals have broad buy in are mapped across the curriculum are mapped to general education outcomes Student work products are rated with institutional rubrics are rated with national rubrics Continuous Improvement at UNCW 1. Monitor annual learning goal process for general education outcomes and courses 2. Make recommendations on curriculum and/or assessment improvement 3. Invite stakeholders across campus to discuss gaps in the data 2
Presentation of Case Studies Case 1: Determining the Long Term Fate of the QEP Case 2: Finding Ways to Close the Loop on SLO and Program Assessment Data Respond to the questions at the bottom of the case study and be prepared to share your responses with the larger group. Our approach to Case Study #1: Determining the long term fate of the QEP UNCW s QEP Reflection Committee December 2016 January 2017 Director, Center for Marine Science (former VC) Director, School of Health and Applied Human Sciences Director, Center for Teaching Excellence and Center for Faculty Leadership Associate Provost, Institutional Research and Planning Associate VC, Business Affairs Chair, Department of Anthropology Professor, Department of Economics and Finance Professor, Department of Computer Science Associate Professor, Department of History 3
UNCW s QEP Reflection Committee Key questions asked by the working group: How has ETEAL influenced student learning? How has ETEAL influenced faculty pedagogy and attitudes toward applied learning? How has ETEAL shaped our institutional processes and approaches? What are the critical continuing applied learning needs on our campus? Data Sources to Assess Impact of QEP 1. Content analysis of syllabi before and after ETEAL 2. Assessment of faculty/student scholarship supported by ETEAL 3. Institutional survey data regarding perceptions of applied learning held by UNCW faculty and staff 4. Review of assessment data from all four years of ETEAL Categories of Results Reported Impact on Student Experience/Learning Impact on Pedagogy Impact on Institutional Processes Need for Continued Support 4
Recommendations in the Report Based on the data analyzed, best practices reflected in the scholarly literature, and our overall sense of campus culture, we make the overarching recommendation that UNCW not just reaffirm, but make a long term commitment to, applied learning. Practical recommendations: Retain or expand current funding and seek external funding as well Add funding to increase number and quality of experiences, showcase leading practices, and share success stories Rename the program and retain a permanent office Increase data collection scope and improve data collection quality Target curriculum related applied learning and high impact practices Incorporate applied learning into best recruitment and student success practices Reward applied learning efforts in workload structures Post QEP Planning Timeline December 2016 January 2017 September 2017 Summer 2018 QEP Reflection Committee forms QEP Reflection Committee report sent to Provost White paper prepared by Undergraduate Studies presented to Provost and Chancellor, offering further details of a proposed rationale, structure and budget for continuing applied learning efforts 5 year QEP period for ETEAL ends Our Approach to Case Study #2: Finding Ways to Close the Loop on SLOs and Program Assessment Data 5
Closing the Loop on SLOs and Program Assessment Data Question from the Chancellor and the Provost: Is there grade inflation at UNCW? Rather than answer with just Yes or No, we took a different approach Collaborative Data Presentation highlighting student behavior: General Education Assessment Results Student Perception Data on academic behavior Student Information System Analysis of Grades and Course Evaluations General Education Assessment Reviewed data from General Education Assessment results Written Communication: Students perform highest on Control of Syntax and Mechanics. They need the most work on Genre and Disciplinary Conventions and Content Development. Students do improve as they progress through their degree programs, but these distributions have not (yet) gone up over time. Information Literacy: Students performed highest on Ethical and Legal Use and performed lowest on Evaluation of Information and its Sources. Through an anomaly in the test group, results show that second year composition students score as high as or higher than all other student groups. One of the most important findings we have made is that the quality of the assignment itself is the most important factor in how well students perform. Student Perception Data Used Student Affairs data on student perceptions related to academic behaviors Confidence in academic ability at various points in First Year Pre College Characteristics Academic Behaviors # of hours plan to study # of hours actually studying Expected GPA 6
First Year Student Perception Freshmen at New Student Orientation indicate they are confident they will succeed academically at UNCW. 98.5% Source: 2016 Freshman Student Orientation Evaluation n = 1,890 First Year Student Perception Freshmen after 1 week of school indicate they are very confident in their academic ability. 86.1% Source: 2016 Beginning Student Survey n = 2,024 First Year Student Perception Freshmen after 5 6 weeks into the academic year indicate they expect their first semester GPA to be a B+ avg or higher (3.3 GPA). 73.8% Source: 2015 Fall Freshman Survey n = 1,903 7
First Year Student Perception First year students 5 6 weeks into the academic year indicate they expect their first semester GPA to be lower than 2.3. < 2% Source: 2015 Fall Freshman Survey n = 1,903 First Year Student Perception First year students 5 6 weeks into the academic year school believe that UNCW s academic standards and expectations are just right for them. 83.4% Source: 2015 Fall Freshman Survey n = 1,903 First Year Student Academic Behaviors First year students 5 6 weeks into the academic year school state they come to every or almost every class academically prepared (have done homework, readings, assignments, etc.). 93.8% Source: 2015 Fall Freshman Survey n = 1,903 8
First Year Student Academic Behaviors Number of Hours/Week 1st Year Students Study 4% 9% 21% 0 5 6 10 25% Source: 2015 Fall Freshman Survey n = 1,903 41% 11 15 16 20 21 or more Grades and Course Evaluations Statistics faculty member Jim Blum used Student Information System (Banner) Data to conduct an analysis of grade inflation Reviewed grades assigned by course, by faculty, over a ten year period Provided visualization of how grades may (or may not) have changed over that period Connected to course evaluations to show the lack of correlation between grade and course evaluation Collaborative Assessment Data 97% of faculty and staff attendees rated the information in the sessions as either Very informative or Informative 88% stated that the information helped to add a broader context for their work 9
What Else Faculty Want to Know Blum pointed out the importance of using the data to illuminate as opposed to using it to uphold a theory. What is being done to ensure that the data collected will be used to illuminate and inform our high level decisions? Is the administration currently using the data in this way? More info on transfer and graduate students I would like to see more information on major/minor students graduation rates when they switch from one subject to the other. I'd be curious to hear the student success expert discuss online, hybrid, and alternative course delivery methods. These course delivery modes seem important in light of the UNCW growth agenda. Closing the Loop Helps us demonstrate that assessment data is useful with proper context Enables recommendations and changes based on the data, rather than on simply anecdote Provides evidence that our programs are working Closing the Loop Other Examples Year of Thinking Critically about Critical Thinking Spring into Writing We ve Heard Your Voice Diversity and Inclusion Teaching Fellows Program 10
Ideas into Action What could you do to take these ideas into action on your campus? Suggestions on how to improve the dissemination of assessment data? What have you tried that worked or didn t work? Q &A 11