School of Behavioral Sciences Counselor Education and Family Studies Ph.D. in Counselor Education Program 2017-18 Annual Report 2016-17 Assessment Cycle Data 1
(Hyperlinks go directly to the identified section of this document) Comprehensive Assessment Plan Procedures...3 Comprehensive Assessment Plan Review Results and Program Modifications...4 Student Learning Outcomes Procedures...6 Results...7 Program Modifications... 10 Reassessment of Prior Modifications... 13 Current Students Procedures... 14 Results... 14 Program Modifications... 15 Alumni Procedures... 16 Results... 16 Program Modifications... 17 Supervisors Procedures... 18 Results... 18 Program Modifications... 18 Employers Procedures... 19 Results... 19 Program Modifications... 20 Demographics Procedures... 20 Results... 20 Program Modifications... 21 Synopsis of Program Modifications... 21 Acknowledgements... 23 Appendices... 24 2
Ph.D. in Counselor Education and Supervision Comprehensive Program Evaluation Annual Report on Assessment, Survey Findings, and Recommendations During the 2016-17 academic years, the Department of Counselor Education and Family Studies conducted extensive evaluations of the Ph.D. in Counselor Education and Supervision program. The Comprehensive Assessment Plan and the Assessment Matrix of Key Performance Indicators (KPI s) for the CACREP Doctoral Student Learning Outcomes provided the procedures that guided the evaluation. We evaluated Key Performance Indicators (KPI s) for the Counselor Education and Supervision Doctoral Competencies across several courses in the curriculum. The KPI assessment rubrics provided an assessment measure for the knowledge and skills deemed essential for our students to master for each of the CACREP doctoral competencies standards assessed by the assignment. Program faculty evaluated each measure used to provide evidence of competence. Once completed, we evaluated the data using LiveText, our assessment management system. Per our Comprehensive Assessment Plan, we also collected data from four groups of stakeholders: recent graduates, current students, practicum and internship supervisors, and employers of our recent graduates. All stakeholders other than current students used a survey format to provide feedback. We collected data from current student data during a series of town meetings with the director of the Counselor Education program during the classroom component of their courses. All other stakeholders responded using a Likert scale survey instrument. Site-supervisors responded through a course assignment in LiveText. We provided alumni and employers with the survey through a direct email contact. Once completed, we aggregated the data, converted aggregate data raw scores (#respondents per ranking) to weighted Likert scale values, and calculated mean Likert scores. The survey return rates were high, with stakeholder samples of 3 of 3 graduates, 8 of 8 site supervisors (fall semester), and all 4 Employers of our 3 graduates. Although this is a limited sample, we were able to evaluate trends in the supervisor, alumni, and employer data as well as aggregate KPI data. We evaluated trends found in 2016-17 assessment data in 2017-18 and used the data to inform current program modifications. We provide the aggregate data for each respondent groups in this report. We then present subsequent program modifications. The discussion concludes with a summary statement and an enumeration of recommendations for improvement. Self-Study Program Evaluation: In addition to 2016-17 assessment data, program evaluation included process of constructing two addendum responses requested by the self-study reviewers as well as preparation for a site visit scheduled for September 2017. Thus, most program modifications for the Counselor Education and Supervision program centered on the doctoral professional identity (6.B.1-5) learning outcome standards and assessment procedures, specifically Key Performance Indicators. Thus, this comprehensive review of our curriculum and the Comprehensive Assessment Plan became the principle evaluation data that informed our program modification this year. Given the importance of CACREP accreditation for our students, we felt that focusing our modifications for accreditation was of paramount importance. 3
Findings: Evaluation of the Comprehensive Assessment Plan In constructing our self-study addendum response in Summer 2017, we consulted with Dr. Jeff Parsons, who helped us resolve the areas where we had incorrectly interpreted the 2016 CACREP Standards (Section 4) for assessment as noted by the reviewers. These areas included: Elimination of the requirement to assess all student learning (Learning Experiences) Addition of representative assessment for the Doctoral Professional Identity Standards (Key Performance Indicators for each of the five doctoral professional identity standards) Addition of our Assessment of Student Dispositions and Student Demographic Data to our Comprehensive Assessment Plan Clearly linking assessment results and program modification to the Program Objectives We evaluated our 2016-17 Comprehensive Assessment Plan in light of these clarifications. We found the need to more clearly define our assessment procedures used for program modification (representative) as opposed to those used to assess individual students progress (retention and remediation) in our Comprehensive Assessment Plan. We present the specific results of our assessment and describe the modifications made to our Comprehensive Assessment Plan below. Shift from the Assessment of All Student Learning: Per our Comprehensive Assessment Plan, we conceptualized Key Performance Indicators as capstone assignments designated as measures of all Professional Identity Student Learning Outcomes (Section 6.B.1-5) and used aggregate assessment data from all assignments to inform program modifications. In other words, we viewed the assignments used to meet this standard as measures of student learning for program evaluation rather than learning experiences used for individual student progress. Furthermore, because we conceptualized Key Performance Indicators as measures, we needed to develop clearly defined Key Performance Indicators as conceptualized in the CACREP 2016 Standards (4.F.1-3) as requested by the reviewers. To meet the standard, we formally identified Key Performance Indicators that our faculty felt best represented the knowledge and skills that all doctoral students must demonstrate for each Professional Identity Standard across the curriculum prior or graduating from the program. Once identified, we designated benchmark assignments across the curriculum to serve as assessment measures for each Key Performance Indicator. Finally, a review of our current benchmark assignments found that all of our designated Key Performance Indicator measures were already a part of our current assessment system. Therefore, making the change to representative assessment required us to 1) update our assessment rubrics into LiveText, 2) add KPI matrices to course syllabi, and 3) change the Student Learning Outcomes matrices to Student Learning Experiences in the CACREP matrices in course syllabi. Once in place, we re-assessed all 2015-16 and 2016-17 data using our KPI rubrics. Furthermore, while we used our individual SLO data for student remediation, it was not a part of our formal assessment plan. Finally, we found that we did not have the institutional and departmental procedures our faculty used to monitor individual student learning in the Comprehensive Assessment Plan. Therefore, the evaluation of our formal assessment plan indicated that we needed to expand our formal assessment of student learning to explicitly include retention and remediation using the assessment of our students learning experiences. 4
Program Modification (Program Objectives #1-5): We revised the Assessment of Student Learning Outcomes in our Comprehensive Assessment Plan to ensure all learning experiences occur and clearly reflect the retention function of student learning outcomes as follows: We modified the curriculum to ensure that we met all required learning experiences. We added the revised Doctoral Competencies Learning Experiences Matrix to the CAP and all course syllabi. We described this process in the SLO Assessment Results in the next section. We also formalized our process for identifying students who do not meet the benchmark grade, both at the assignment level (Submission and Success Rates) and course level (At-Risk Report) and added it to the Comprehensive Assessment Plan. This allows us to determine the nature and level of the academic deficiency (knowledge; skill; disposition), and to help us to develop an action plan as indicated for the purposes of retention and remediation. We also made modifications in our Comprehensive Assessment Plan to ensure that our assessment of student learning outcomes for the purpose of program evaluation clearly reflect the use of our data to inform modifications to our program objectives as follows: We added the Assessment of Program Objectives section to the Comprehensive Assessment Plan, which focuses on the program evaluation process for assessing Program Objectives. In this section, we clearly link our representative assessment to the program evaluation requirements outlined under CACREP Standard 4.B and 4.F. We developed Key Performance Indicators for each of the five doctoral professional identity (Section 6.B) standards that we use to evaluate and modify Program Objectives. We identified benchmark measures for each Key Performance Indicator at multiple points throughout the program. Then, we constructed the KPI Assessment Measures Matrix to define the measures used for representative assessment of student learning. We added the KPI Assessment Matrix to the Comprehensive Assessment Plan as well as to each course that contained a KPI measure. We added the KPI Curriculum Map to the Comprehensive Assessment Plan We linked all Key Performance Indicators to our Program Learning Outcomes, which shows how we use our KPI data to assess student learning to support our Program Objectives and subsequent program modifications. Student Dispositions: Each course instructor filled out the Professional Development Rubric for every doctoral student in every course taken in the program each semester. This allowed us to monitor individual student s disposition across a variety of courses under different instructors. We had a system in place to assess dispositions and monitored students for the purpose of remediation, but we did not include dispositions in our formal assessment plan or Gates 2, 4, 5, 6, and 7 reviews. Program Modification (Program Objectives #1-6): In August 2017, we added the Assessment of Student Dispositions section in our Comprehensive Assessment Plan. This outlined a formal plan to assess student dispositions for the purposes of individual remediation as well as program modification as we seek to improve our ability to produce competent ethical counselors and counselor educators. We also included a requirement that students must meet competencies on the Professional Development Rubric as a part of Gates 2, 4-7. 5
Assessment of Student Demographics: While we had a process in place to collect demographic data for our current students for the Vital Statistics Report, we did not have a formal process of collecting this data for the three groups of students: applicants, current students, and graduates. Thus, our Comprehensive Assessment Plan did not include an assessment procedure required under the 2016 Standards. Program Modification (Program Objectives #1-6): In Summer 2017, we added the Assessment of Student Demographics section in our Comprehensive Assessment Plan. This outlined procedures used to assess current, alumni, and applicant student demographic data as we seek to improve our ability to recruit and maintain a diverse student population as a function of program evaluation. Summary of Program Modifications: Comprehensive Assessment Plan: Maintained the initial system of Gates and our overall system of assessing data, but added dispositions to the check-lists. Clarified the assessment processes informing individual student remediation from those that inform program evaluation as seen in the Assessment of Student Learning and Assessment of Student Dispositions sections in our Comprehensive Assessment Plan. We added the Doctoral Curricular Learning Experiences Matrix. Clearly defined our representative assessment of all doctoral professional identity standards for the purpose of evaluating program outcomes in the new Assessment of Program Outcomes section, including the KPI Matrix, and Curriculum Map Categorized three groups of stakeholder (Alumni, Site-Supervisors, and Employers) under a new Assessment of Community Outcomes section, to reflect the importance we place on the assessment data we receive from sources outside of the department. Clearly outlined our general procedures for using assessment data and subsequent modifications to programs, course, and policies under the Using Findings for Program Modifications section of the Comprehensive Assessment Plan. Added a Summary of the Departmental Assessments to clearly outline all aspects of our Comprehensive Assessment Plan in an aggregated format. Findings: Student Learning Outcomes We evaluate students in the program throughout their studies for their benefit and to ensure that program graduates are prepared to successfully participate in the profession of counseling and counselor education. This process includes a series of evaluations outlined in the Comprehensive Assessment Plan (CAP). The CAP defines the benchmarks used to assess Counselor Education and Supervision students academic, professional, and personal development as they progress through the program. In addition, the Key Performance Indicators (KPI) Matrix allows us to identify every course and assignment containing KPI measures. Finally, each syllabus contains a CACREP KPI Assessment Matrix listing the KPI assignments associated with the course and the Doctoral Student Learning Outcomes matrix, which links 6.B standards to course assignments. This provides course instructors with the CACREP standards for each course assignment and provides guidance when evaluating students for mastery in the course. These matrices provide the foundation for our assessment of student learning, both at a program and individual level. 6
LiveText is the assessment management system used to assess, aggregate, and analyze the data generated from the Key Performance Indicators and Professional Dispositions rubrics. This system provides evaluative data that can inform program modifications as well as identify individual students who do not meet requisite competency levels. In addition, we use the Field Evaluation Module (FEM) in LiveText for all supervision-based assessments used in the Practicum and Counseling Internships. Assessment Procedures: For the 2016-17 academic year, we used the Key Performance Indicator rubrics to assess student learning for the purposes of program-level evaluation. Faculty assessed mastery of the competencies associated as outlined in the KPI rubrics for all designated assignments. KPI data was analyzed using the report tool in LiveText, the findings were reviewed, and potential action plans to improve learning outcomes were determined. This process allowed us to effectively manage our program learning assessments. In addition, given our upcoming site visit, we did a final evaluation of our program courses as a part of the final accreditation process. We reviewed all course syllabi with the following goals: 1) ensure that all learning activities designated learning experiences and assignments used as measures for KPI s accurately represented our expectations for our students; 2) identify learning activity descriptions that needed to be revised to make the learning experience more explicitly tied to a standard; 3) identify weak learning experiences where our student would benefit by adding or revising learning activities; and 4) identify counselor education-specific learning activities that bridged the five doctoral competencies, including counselor/counselor educator identity and professional development. Once completed, we constructed an updated Doctoral Competencies Learning Experiences Matrix to identify all course learning activities that met each standard. We systematically reviewed the matrix to identify any Section 6.B standard we felt could benefit from additional or revised learning experiences associated with each standard. Reassessment: A key aspect of our assessment of Student Learning Outcome assessment is to examine the effectiveness of prior program modifications. For the 2016-17 assessment cycle, we examined the courses revised in 2015-16. Upon completion of the course, the faculty (SME, course instructors, and/or the assessment coordinator) reviewed the students performance in meeting the competencies measured by the new or revised assignments. After faculty evaluated the effectiveness of these assignments in meeting our learning outcomes, we created another action plan if needed. Results: Student Learning Outcomes Prior to analyzing the KPI assessment data, we established the following target: 90% of the students will rate a competency level of 3.0 (met) or better as benchmarks. In reviewing the findings for our KPI assessments (informing PLO s #1-5), we exceeded our target for each key performance indicator we assessed: 91% to 100% of our Counselor Education and Supervision students met or exceeded expectations across all measures of learning. We provide the Counselor Education and Supervision Program Objectives, Key Performance Indicators, Rubric Items, and Measures used in our assessment of Student Learning Outcomes in Appendix A. Below are the Key Performance Indicators for the five doctoral professional identity competencies: 7
Counseling (Program Objective 4): Students will demonstrate the ability to critically evaluate and apply counseling theories across diverse populations and settings. Key Rubric Competencies: The students will be able to: 1. Critically analyze a broad range of relevant counseling theories 2. Synthesize counseling theories into a comprehensive bio-psychosocial-spiritual theoretical approach 3. Conceptualize a case consistent with the student s theoretical approach 4. Develop a treatment plan consistent with their theoretical approach, 5. Develop a treatment plan that reflects best practices and/or evidence-based interventions for diverse populations Supervision (Program Objective 1): Students will demonstrate the ability to supervise master s level counseling students in an ethical and culturally sensitive manner within a defined theoretical approach Key Rubric Competencies: The students will be able to: 1. Develop a personal theoretical approach or model of supervision 2. Apply their personal model or theoretical approach to their supervision of entry-level practicum students 3. Provide competent and ethical group supervision to entry-level practicum students 4. Demonstrate knowledge associated with cultural competence in supervision practice 5. Adapt supervision to meet the individual differences, learning styles, and developmental stage of the supervisee Teaching (Program Objective 2): Students will demonstrate the ability to apply teaching methods and models of adult learning relevant to counselor education course preparation and delivery Key Rubric Competencies: The students will be able to: 1. Articulate and apply a personal philosophy of teaching to course delivery 2. Deliver competent instruction in an entry-level core counselor preparation course 3. Design a course syllabus for an entry-level core course aligned to CACREP standards, 4. Demonstrate a respect for and respond to individual differences, learning styles, and developmental stage of students Research (Program Objective 3): Students demonstrate the knowledge and competency in research methodology, execution, and dissemination necessary for conducting doctoral level research Key Rubric Competencies: The students will be able to: 1. Demonstrate knowledge of research design, including appropriate sequence of activities involved in research project development. (Knowledge) 2. Design and implement a qualitative or quantitative research study relevant to the field of counseling and counselor education (Skill) 3. Identify and respond to ethical and legal dilemmas associated with research involving human participants. (Knowledge) 4. Demonstrate the ability to produce a scholarly manuscript in a peer-reviewed journal and/or a proposal for presentation at an ACA-affiliated conference (Skill) 8
Leadership and Advocacy (Program Objective 5): Students will demonstrate understanding of theories and skills of leadership and the roles and responsibilities of counselors and counselor educators in leadership and advocacy Key Rubric Competencies: The students will be able to: 1. Articulate a personal philosophy of leadership in counselor education 2. Articulate the role of counselor educators in professional advocacy and leadership in advancing the field of counseling 3. Demonstrate intentionality in developing a self-reflective social justice action plan consistent with the multicultural competencies 4. Demonstrate an active involvement in the profession through licensure, scholarly activities, advocacy, and social justice activities The analysis of our Student Learning Outcomes (Key Performance Indicators) results are as follows: Competency Rubric Exceeded Met Percent Met Emerging Item Competency Competency Target Scores Counseling 1.1 5 5 1 90.9 1.2 5 6 0 100 1.3 5 6 0 100 1.4 5 6 0 100 1.5 5 6 0 100 Supervision 2.1 34 20 0 100 2.2 31 22 0 100 2.3 26 26 0 100 2.4 22 29 0 100 2.5 45 9 0 100 Teaching 3.1 16 40 0 100 3.2 16 40 0 100 3.3 16 34 0 100 3.4 16 40 0 100 Research 4.1 10 25 2 94.6 4.2 10 25 2 94.6 4.3 8 29 0 100 4.4 11 24 2 94.6 Leadership 5.1 4 25 1 96.6 5.2 34 11 0 100 5.3 15 15 0 100 5.5 5 5 0 100 Evaluation of Courses/Syllabi Results: In addition, based on the four criteria examined during our 2016-17 syllabus and course review, we found that our assignment instructions accurately reflected our expectations for learning (Criteria #1), assignments were explicitly tied in to the standard addressed (Criteria #2), and diverse learning activities in the area of counselor educator 9
identity (Criteria #4), including a strong counselor identity (Criteria #4). However, we found some areas where our student would benefit by adding or revising learning activities (Criteria #3), particularly in leadership and the professional development areas expected of future core faculty (CACREP 1.X.3-4, 6.B.4-5). We also found that while the outcomes were positive on changes made in the doctoral internships, we found that providing three separate internships given the current six credit hours made it difficult for faculty and students to navigate their internship experiences (CACREP 6.C). As a result of this review, we based our strategy for program modifications on the following criteria (with related Program Objective): 1) Strengthen counselor education identity, particularly in leadership and advocacy (PLO #5); 2) Develop competencies required of CACREP core faculty for professional development, participation in the field of counseling, and research activities (PLO #3); 3) Further differentiate competency-specific doctoral internships (PLO #1-5), and 4) Adjust the Degree Completion Plan to accommodate changes made in the curriculum to reflect greater professional development of our students as counselor educators. In summary, students met or exceeded our targets for all measured performance indicators we designated to inform program modification supportive of our Program Objectives. Our review of the curriculum and reassessment of prior program modifications found that we met all target scores and standards. However, given the program s goal to train leaders and future core faculty, we felt that students would benefit from a stronger, more focused, approach in their training to be leaders, advocates, and disseminators of research in the field of counseling and counselor education. The program would also benefit from creating separate internship courses to reflect the distinct nature of each internship experience. As a result of these findings, we made the following program modifications in order to further refine and strengthen our current curriculum: Program Modifications: Counselor Educator Identity (Strategy #1-2): A review of the syllabi found that we had developed effective learning experiences where students demonstrated competencies for the intended doctoral outcomes. However, we believed that further modification of the curriculum would help our students to be better prepared to fulfill the expected activities of a counselor educator in an academic setting. Therefore, we made several curricular changes, replacing some of the more clinically focused courses in the program with courses that would specifically develop a strong counselor educator identity. Therefore, we made the following modifications to the Degree Completion Plan based on the Fall 2016 curricular review for the 2016-17 assessment cycle: 1) Addition of Counselor Education Competency Courses (Program Objectives #3, 5): While we had courses specific to four of the professional identity competencies, we did not have a course specific to the Leadership and Advocacy competency. Furthermore, given the CACREP (1.X.3-4) research, professional identity, advocacy and service, and professional development requirements for core faculty, we wanted to strengthen our students learning experiences in order to further develop these skills. To enhance learning, we added the following two courses to the curriculum: 10
Dissemination of Research and Scholarship (Program Objective #5): In order to give our students a voice of impact in the field of professional counseling, they needed to develop expertise in developing and disseminating scholarly professional counseling presentations, research and conceptual manuscripts, newsletters, grants, etc. The addition of COUC 810 Dissemination of Research and Scholarship in Counseling course to our curriculum will ensure that our students have a strong foundation in writing and disseminating presentations and publications to the field of counseling in a variety of venues, the principles of research and scholarly writing, and readiness skills for manuscript preparation, submission, review, editorial, and presentation processes. We believe that this course s comprehensive approach strengthens our students competence in the professional scholarship required for future core faculty. We will assess the effectiveness of the course during the 2017-18 assessment cycle. Leadership and Advocacy (Program Objective #5): Given our commitment to train leaders in the field of counseling and counselor education (PLO 5), we felt that students needed a stronger, more focused, approach in their training to be leaders and advocates in the field of counseling and counselor education. The addition of COUC 860 Leadership and Advocacy in Counselor Education to our curriculum ensures that our students have a strong foundation in leadership and advocacy by fostering their professional identity as a counselor educator through leadership and service to the counseling profession. We believe that this course s comprehensive approach strengthens our students competence in leadership and advocacy. We will assess the effectiveness of the course during the 2017-18 assessment cycle. 2) Differentiation of Internship Courses (Strategy #3): In 2016-17, we implemented the following program modifications: 1) All CES students are now required to take a Teaching Internship and a Supervision Internship. For their third competency areas, students who are not licensed must take the Counseling Internship; however, students who are licensed have the option to take either a Counseling, Leadership/Advocacy, or Research Internship. We reassessed the effectiveness of these changes during the current assessment cycle. Assessment of student learning for the revised COUC 999 Counseling Internship courses taught during the 2016-17 academic year found that students met or exceeded competence in all KPI measurements. In addition, student and faculty feedback indicated a high level of satisfaction with the new internship structure. While the results suggest that the change was beneficial, we found some logistical issues that still created challenges for faculty and students. These include: 1) Using the same course number for all internships made it difficult to schedule one course for three separate internships. 2) It was challenging to manage three distinct semester-long internships using two required courses. This required us to manually track the students internships to document the completion of three separate field experiences. Finally, students reported that they found it confusing at times to navigate the logistical and scheduling aspects of using singular course for three distinct internships. To address the concerns that emerged from our reassessment of the doctoral internships, we made several changes for the 2017-18 Degree Completion Plan (Program Objectives 1-5). 1) We added two courses to the curriculum: COUC 970 Teaching Internship and COUC 980 Supervision Internship. This supports our requirement that all doctoral students take a 11
Teaching Internship and a Supervision Internship; 2) We changed COUC 999 Internship to the Counseling, Leadership/Advocacy, Internship, and/or Research Internships; 3) Because we added COUC 970 and COUC 980 to the DCP, we reduced the required hours of COUC 999 to three (3) hours. This increased the required number of internship-related credits from six to nine credit hours. Students are now required to take a total of nine internship-related credit hours. 3) Addition of Professional Practice Courses (Strategy #3-4): To reflect the shift from a clinical mental health counseling approach in the internships to experiences based on three of the five professional identity competencies, we moved the practicum and internship courses to this new classification on the Degree Completion Plan. In addition, as noted above, we added the following to the section titled Professional Practice Courses (12 Credit Hours): COUC 998 Doctoral Practicum COUC 970 Teaching Internship COUC 980 Supervision Internship COUC 999 Doctoral Internship (Counseling, Research, or Leadership/Advocacy) 4) Re-Designation of Select Clinical Courses (Strategy #3). To accommodate the addition of the above nine credit hours to the curriculum, we evaluated all courses in light of their effectiveness in meeting Program Objectives 1-5, which is to produce counselor educators who are competent in the five doctoral professional identity areas. To this end, we made the following changes in the Degree Completion Plan: Core Courses: We changed the following courses from required courses to electives: COUC 720 Advanced Family, Systems & Development COUC 800 Advanced Assessment Advanced Clinical Mental Health Courses (9 Hours): We reduced the number of advanced clinical courses from 21 Hours by moving the practicum and internship courses (9 hours) to the Professional Practice section. We also changed the following courses from two required courses (6 hours) to students selecting one of the two courses (3 hours). COUC 815 Empirically Supported Treatments for Adults COUC 820 Empirically Supported Treatments for Children & Adolescents Dissertation Courses: In our pre-dissertation research design course, students selected a quantitative (COUC 870) or qualitative (COUC 871) methodology for a potential dissertation research design. However, we felt that our students would benefit from exposure to a variety of research designs in the early stage of the dissertation process. Therefore, we designed the COUC 870 Advanced Research Design course to ensure that students had exposure to a variety of methodological and practical issues involved in research designs and advanced data analysis procedures. By presenting their own dissertation research ideas, students will be able to craft research questions and select appropriate research designs and analytic strategies for a variety of research relevant to the field of counseling and counselor education. To accommodate this change, we removed the following select one of the following courses, replacing them with the revised COUC 870 Advanced Research Seminar course. 12
COUC 870 Quantitative Research Seminar COUC 871 Qualitative Research Seminar In summary, in order to strengthen the counselor educator identity and maintain the required number of credit hours to graduate from the program, we evaluated the courses required on the 2016-17 Degree Completion Plan. We then made the above program modifications to the program s 2017-18 Degree Completion Plan. We will assess the effectiveness of these changes in the 2017-18 assessment cycle. Reassessment of Courses Modified 2016-17 COUC 710 Advanced Group Counseling: The 2015-16 program modifications implemented during 2016-17 centered on group leadership skills, application of advanced group theories, and implications for working with master s students. We reassessed the course student learning outcomes to evaluate the effectiveness of these changes. Assessment of the Student Learning Experiences in the course taught during the 2016-17 academic year found that students met or exceeded competence in each measurement. COUC 715 Advanced Theory Application: The 2015-16 program modifications implemented in 2016-17 centered on a stronger emphasis on ethical and cultural competence with a diverse population when applying theories and practical application of advanced theory to their roles as counselor educators and supervisors. We reassessed those course student learning outcomes to evaluate the effectiveness of these changes. Assessment of the Student Learning Experiences in the course taught during the 2016-17 academic year found that students met or exceeded competence in each measurement. COUC 747 Instruction in Counselor Education: The 2015-16 program modifications implemented in 2016-17 centered on a stronger counselor education approach, specifically in regard to course design, delivery, and course evaluation methods appropriate to counselor education learning outcomes. We also added taped practice sessions, with peer and self-reflective feedback, within the course. These modifications are consistent with employer feedback. Assessment of the Student Learning Experiences in the course taught during the 2016-17 academic year found that students met or exceeded competence in each measurement. COUC 815 Empirically Supported Treatments for Adults: The 2015-16 program modifications implemented in 2016-17 centered on adapting assignments to better reflect the CACREP standards, promoting the formation of a strong counseling / counselor educator identity, increasing awareness of and attention to ethics and multicultural competency in treatment planning as a counselor and counselor educator, and implications for teaching or supervising a master s-level student. We reassessed the course student learning outcomes to evaluate the effectiveness of these changes. Assessment of the Student Learning Experiences in the course taught during the 2016-17 academic year found that students met or exceeded competence in each measurement. Note: This course is now an elective in the program. 13
Stakeholder Survey Findings and Recommendations Current Students: Focus Group Data CES doctoral students participated in a series of town hall meetings moderated by the program director. These groups were held during the classroom portion (intensive week) of their courses. We asked students about their satisfaction with all areas of the program and gave them the opportunity to ask questions and express their concerns directly to the program director. This format allowed the department to identify students concerns, with the intention of addressing them. Some areas that we specifically asked students for feedback included: a) quality of preparation of the counselor education skill and knowledge areas; b) structure and delivery of the curriculum; c) program advising and support; d) practicum and internships; and e) operational structure of the program. Results: During these meetings, the students focus was primarily on the status of our CACREP accreditation process, expressed the importance of accreditation to them, and shared about their experiences in the program during the program s alignment to the 2016 CACREP standards. In terms of curriculum, students expressed overall satisfaction with and appreciation of the content and delivery of the courses. Although the intensive portions posed challenges to those students not within commuting distance to Lynchburg, all expressed that this aspect of the program was important and valuable to them (e.g. They are worth the expense and time away ). Some students expressed confusion about recent program revisions, including the addition of a third internship. Others were anxious about fulfilling the requirements of the Candidacy Portfolio, in particular those items on the checklist that were not course dependent. There were also questions about how to best organize and present the portfolio. Students in the CES program reported feeling connected to one another and to department faculty. Overall, students find the faculty warm, engaging, responsive, and helpful as mentors. A few distance students expressed feeling like they were not as connected to peers or faculty as they would like to be. Questions revolved around how to go about getting more involved with faculty and peer research and scholarship activities. In terms of advising, some students were still confused about the role and function of their faculty advisors, including when they should contact faculty and inconsistency in the quality of their experiences. Students reported that the process of course registration and academic aspects of advising was generally positive. We received approval to induct Ph.D. students into CSI, but some students were still not aware that they could serve in leadership positions in our CSI chapter or in CSI at large. Additionally, they had questions about how to get more involved in ACA and ACA division leadership at the local, state, and national levels. Students reported feeling positively about the intensive facilities (the Liberty Mountain Conference Center) and the opportunities for connection provided during the snack breaks and luncheons. Students shared that during the intensives they enjoyed the collaboration and connection that occurs, appreciative that it leads to meaningful post-intensive opportunities for collaborative research and scholarship. 14
Program Modifications: Key areas addressed by students in the 2016-2017 town meetings centered on concerns about CACREP accreditation and related program revisions, including aspects of the Candidacy Portfolio and faculty advising. A few expressed apprehension about not hearing back quickly enough from academic advisors, the support coordinator, and certain faculty members. Some students reported wanting more connection with peers and faculty, particularly research and scholarly collaboration. Finally, some students wanted more information about serving in leadership positions in CSI and ACA/ACA divisions. CACREP Accreditation (Program Objectives #1-5): Student feedback over the past year consistently stressed the importance of the department seeking CACREP accreditation for the Ph.D. in Counselor Education and Supervision program. Given the strong student support for accreditation, we accelerated our response to the addendum requested by the reviewers of the self-study. As a result, CACREP approved a site visit for September 2017. We also found that it was very important to keep students informed of the rationale for the multiple program modifications that we made during the process of as aligning the program with the 2016 standards. Therefore, we made a point to explain our program modifications in terms of CACPEP standards during our town meetings with the students. Advising: To address concerns about advising, we added an advising component to the town hall meetings held during each intensive. The program director, along with program faculty, address the concerns of CES students during each intensive course as well as throughout the year. During these meetings students are encouraged to express their advising needs as well as any concerns they have about the program. This allows us to work collaboratively with students. We also clarified that students may contact the program director, faculty advisors, peer mentors, and program faculty if concerns arise between intensives. To support this, we oriented program faculty to their roles as faculty advisors, including their mentorship responsibilities, the logistics of advising students through the Candidacy Portfolio process, and the requirement that advisors respond to advisee s e-mails within 24-48 hours. Faculty also regularly reach out to students to inform them of their advocacy, leadership, research, and scholarship endeavors and invite them to join them in these. Leadership Development (Program Objective #5): To help orient students to their professional identity as leaders in the field of counseling, the program director now contacts students upon acceptance to the PhD CES program and assigns a peer mentor. At town meetings, we highlight the importance of joining CSI, ACA and ACA divisions related to their specialty area, and state ACA affiliated organizations. We also introduce and promote opportunities for leadership and advocacy, as well as collaborative research and scholarship opportunities, through the CES Center. Connection (Program Objective #5): The CES Center serves as the central information hub for all CES students. The program director and faculty and academic advisors remain in continual contact with students through the CES Center. In addition, we use the center to connect new students with a peer mentor who is further along in the program through the peer mentorship program. The mentor serves as a conduit for building connections to others in the program, 15
inviting new students to engage in collaborative projects, leadership activities, and other meaningful opportunities to connect with peers, faculty, and the counseling community at large. Alumni: Survey Data In Spring 2017, we received the results of the Office of Institutional Effectiveness Alumni Surveys that were sent to recent Liberty University graduates in October, 2016. At that time. we discovered that there was no data available from the 3 students who graduated from the Ph.D. in Counselor Education and Supervision program. When we followed up with our graduates, they did not recall receiving a survey request. Therefore, we made the decision to send our Alumni Survey directly to our alumni. In February 2017, we revised the CES-specific survey questions to better reflect the data we wanted to collect from our graduates. Once reviewed, we emailed the new survey to alumni to the three graduates of the Ph.D. in Counselor Education and Supervision program. In Section 1, alumni used a dichotomous scale to rate if the preparation provided was sufficient and if the course was relevant for each course on the Degree Completion Plan (DCP): Yes or No (N/A if they did not take the course). In Section 2 and Section 3, alumni rated their personal satisfaction for each item using a 5 point Likert Scale, Alumni responses ranged from 5 - Very Satisfied to 1 Not at all Satisfied; (0: N/A). Finally, we asked alumni open ended questions concerning strengths, weaknesses, general feedback, and areas for improvement. All three graduates (100%) of the Ph.D. in Counselor Education and Supervision program returned the survey (100%). Once we received the results (minus identifying information), we aggregated the data, converted aggregate data raw scores (#respondents per ranking) to weighted Likert scale values, and calculated mean Likert scores using Excel. Results: Overall, alumni were very positive in their perceptions of the program. With respect to the structure and relevance of the curriculum, graduates reported that the preparation provided in each course taken was sufficient and relevant to their roles as counselor educators. In evaluating their personal satisfaction with their program in relation to their competencies in the five CACREP Doctoral Competency Area Student Learning Outcomes, alumni reported being Satisfied to Very Satisfied in the following doctoral competency areas: Counseling, Supervision, and Teaching (4.7 to 5.0 out of 5). Areas rated Satisfied in these competency areas included: integration of counseling theories relevant to counseling (4.7); culturally relevant strategies in counseling (4.7), supervision (4.7), and teaching (4.7); administrative functions in supervision (4.7); and remediation/gatekeeping functions in teaching (4.7). In evaluating their personal satisfaction with the Research competency, students reported being Moderately Satisfied to Very Satisfied (3.7 5.0). Areas rated Moderately Satisfied mainly centered on standards concerning the dissemination of information as a counselor educator: methods of program evaluation (4.3); research questions for professional research/ publications (4.0); professional writing (4.3); conference proposals / presentations (3.7); and grant proposals and funding resources (3.7). Alumni also noted being Moderately Satisfied to Very Satisfied (4.3 5.0) with the their competencies in the Leadership / Advocacy competencies. Standards that were rated Moderately Satisfied to Satisfied mainly centered on the administrative functions of leadership, such as consultation (4.0) and crises and disasters (4.0). Other areas rated Satisfied 16
centered on the standards addressing advocacy: models (4.3); political issues (4.3); and ethically and culturally relevant strategies of leadership and advocacy (4.3). When asked to evaluate the non-curricular aspects of the program, alumni expressed the greatest satisfaction with the overall faculty advisement, competence, availability, feedback, and assistance in their professional development; supervised field experiences; instructional resources; and opportunities for advanced professional development (4.7 5.0). Graduates were slightly less satisfied with the faculty s scholarly productivity (4.3); opportunities to collaborate (4.0) or exposure to ongoing research (4.3) with faculty. Alumni s suggestions for improvement centered on peer support (i.e., cohort) while in the program. One graduate noted that he or she would have enjoyed more opportunities to connect with the cohort, while another stated Because students come in from all over the country for one intensive week at a time, it is easy to feel disconnected and isolated anything that supports a sense of comradery and cohort is definitely a positive move to increase peer support through such a challenging program. Finally, one alumnus suggested more emphasis on the development of teaching and supervision methodologies in the program. In summary, our graduates were very positive in their perceptions of the program. While our students were Very Satisfied (74% of all responses), reviewing the Satisfied (21%) and Moderately Satisfied (5%) responses provided areas for further examination. These appeared to center on learning more culturally relevant strategies across the core competency areas, administrative and gatekeeping aspects of leadership, advocacy, greater involvement in all aspects of faculty research (e.g., collaboration, design, mentoring, dissemination of research results), and a greater sense of connectivity with other students in the program. Note: We recognize that the very small number of alumni (N=3) from the program presents a challenge to making substantive changes based on this data. However, we found that areas that emerged for program modifications are consistent with results of our current student survey and the curriculum review from our assessment of student learning. The data supported the following program modifications: Program Modifications Professional Development (Program Objectives #3, 5): Modifications supported by our graduate survey include the addition of COUC 810 Dissemination of Research and Scholarship and COUC 860 Leadership in Counselor Education to the Degree Completion Plan. In addition, we instituted the Doctoral Professional Identity Competencies, which require student involvement in ACA and ACA affiliated professional organizations through membership and participation in conferences, presentations, and publications, and a commitment to licensure. Faculty and Peer Collaboration (Program Objectives #3, 4, 5): Given the increased focus on research and scholarship skills required of future core faculty (CACREP Standard 1.X.3.a-c), doctoral students are able to engage in greater collaboration with faculty research projects. Finally, we now centralize the dissemination of information to students through the Counselor Education and Supervision Center. The center provides a venue for announcing opportunities for faculty mentorship, teaching, and other professional development experiences, and voluntary 17
peer mentoring for students wanting a greater level of connectivity with other doctoral students. We believe that these enhancements to the CES Center will facilitate a greater sense of community among faculty and students. Clinical Site Supervisors: Survey Data We embed the Site-Supervisor Survey as a course assignment in the Field Experience Module (LiveText ), which we use to manage evaluation rubrics for clinical practicum and internships. We ask site supervisors to submit the survey along with their final evaluation of the student. Participation is voluntary, and we ensure that faculty or students cannot view the supervisor survey results. The following number of supervisors participated in the site supervisor survey in Fall 2016: Practicum: (N=5); Internship: (N=3). Practicum Overall, on-site supervisors are positive in their evaluations of how we prepare our practicum students for the field experience component of counselor education training. Site supervisors indicated that our program was effective in producing students who were well-prepared in the areas of professional ethics, identity, professional behaviors and dispositions (Average of means: 4.0 out of 4). Supervisors also felt we produced students who were adequately to well-prepared in the skills (3.8 4.0) and knowledge (3.8 4.0) in the core content areas. While the overall results are very favorable, we continually strive to enhance our students preparation for their clinical experiences. Internship Overall, on-site supervisors are positive in their evaluations of how we prepare our internship students for the field experience component of counselor training. Site supervisors indicated that our program was effective in producing students who were well-prepared in the areas of professional ethics, identity, and dispositions (Average of means: 3.8 of 4) and adequately prepared in involvement in professional organizations (3.3 of 4.0). Supervisors also felt we produced students who were well-prepared in 10 of 12 skill areas and 10 of 12 knowledge areas (Average of means: 3.74 of 4) and adequately prepared in their knowledge and skills in data analysis and applied research (3.3 of 4.0). Program Modifications The overall results are favorable. Unfortunately, due to the very small sample size (N=3), it can be a challenge to make substantive interpretations of the data. However, we feel that feedback on even a single student provides an opportunity for us to better prepare our students for practice as a counselor educators. Areas that stood out for program modifications in the data included greater focus on data analysis, applying research to practice, and issues associated with professional development. Given the result of the 2016-17 supervision and student assessment results, we made the following program modifications in 2017-18. Professional Development (Program Objectives #3, 5): We addressed the areas of professional development through the addition of the Professional Development Competencies requirements as a graduation requirement as well as the addition of COUC 810 Dissemination of Research & Scholarship in Counseling into to the Degree Completion Plan. We also adapted an assignment 18