Assessment of Student Academic Achievement

Transcription

1 Assessment of Student Academic Achievement 13 Chapter Parkland s commitment to the assessment of student academic achievement and its documentation is reflected in the college s mission statement; it also advances the college s six values and these purposes: To provide technical-vocational and career education that meets the needs of students, business, and industry. To prepare students for transfer to four-year colleges and universities. To provide developmental programs, courses, and services which prepare students to pursue collegelevel work successfully. To develop and enrich students general education, including effective written and oral communication; mathematical, scientific, and computer literacy; critical thinking; creativity; and a recognition of the cultural value of history, geography, literature, music, and art. Institutions scheduled for North Central evaluation from 1991 through spring 1995 were expected to show progress toward establishing a plan for assessing student academic achievement but were not required to submit a formal plan. Parkland s preliminary academic assessment work had begun in 1991, when the assessment advisory committee identified the relationship between assessment and the college s mission and purposes in order to construct a foundation for assessing student learning. When Parkland s North Central comprehensive evaluation took place in 1993, the visiting team suggested that we accelerate the development of a comprehensive plan for assessing student academic achievement but that we take time to identify all of the learning outcomes needed to fulfill Parkland s mission and purposes. Please see Table 13.2, Academic Assessment Time Line , at the end of this chapter. Formed in 1994, the 19-member academic assessment committee (AAC) operates under the constitution of the PCA. Its first task was a complete review and analysis of the college s statement of mission and purposes in order to develop an assessment process that would assure that those promises are kept. Faculty from each academic program chose the relevant purpose statements, and then went on to develop goals to support them. The department chairs were introduced to the academic assessment process, so they could assist their faculty. The next step was to identify direct and indirect measures of student learning. Direct measures, such as portfolio assessments, standardized tests, licensure, certification on professional exams, and facultygenerated tests, were identified. Indirect measures that assess students and others opinions of their learning were identified, such as graduate and employer surveys, faculty surveys, graduate exit interviews, focus groups, graduate follow-up studies, graduation rates, transfer rates, and job-placement data. Program faculty developed specific criteria for assessing student performance, which involved writing the ideal product or observable behavior expected of students after instruction. The faculty then developed measurable objectives of student learning that are directly linked to program goals. These objectives specify the measures, methods, and criteria used to assess student behavior, focusing on what students will know, do, and/or think. The first time the assessment cycle was implemented, baseline data were collected. OIRE, which serves the college s research and planning needs, helped identify which groups of students or former students would be analyzed for data collection. From this information, benchmarks were determined after comparing the baseline data to national norms and the faculty s professional recommendations. (Each time new data are obtained, they are compared with previous data to determine whether the criteria have been met, not met, or exceeded. The faculty determine the value of the information and make recommendations for refining the assessment process for their program.) The program faculty then consider which variables contributed to student learning outcomes, such as curriculum design, faculty teaching styles, measures of learning, students behavior, classroom activities, and student learning styles. At this point, the program faculty, with help from the department chair, determine the changes needed to improve learning and develop an action plan that includes a time line, a budget, and both faculty and student responsibilities for changes. Where applicable, improvement and changes may be linked to the college s operational planning process. Figure 13.1 describes the academic assessment cycle. ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT 109

2 Figure 13.1: ACADEMIC ASSESSMENT CYCLE Source: Cecilia López/AAC Assessing Students Learning in Parkland s Academic Programs A. Foundation of the Process 1. Review and analyze Parkland College s mission and purposes statements. B. Overview of the program s purpose and goals 2. Develop and publish the program s mission and purposes. 3. Develop and publish the program goals. F. Action 16. Retain or revise instructional methods and suggest changes in student learning strategies, student academic services, and learning resources as needed. 17. Incorporate the proposed changes into the planning and budget process. 18. Incorporate changes into the five-year review process. 19. Monitor and evaluate the effectiveness of specific changes. 20. Revise the program s mission and purposes, goals, and objectives as needed. 21. Contribute to the revision of Parkland College s mission and purposes as needed. C. Methods/criteria/intended outcomes (objectives) 4. Identify methods for assessing, including appropriate measures. 5. Establish criteria. 6. Develop learning objectives. 7. Identify populations to be assessed. E. Analysis 11. Compare new data with the data collected in previous assessments. Determine whether criteria have been met, not met, or exceeded. 12. Share data with the program faculty and the department chair. 13. Determine the variables that contribute to student outcome. 14. Plan for changes to improve learning. Celebrate success. 15. Share information gained with community groups. D. Data gathering/results 8. Administer the measures. 9. Establish baseline data. 10. Summarize the results. Provide graphs and charts to illustrate the results. 110 ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT

3 The academic assessment process works only if the insights gained spur action. Findings may require a change in teaching strategies, in student learning strategies, in the availability of academic support services, or in a combination. Proposed changes are incorporated into the planning and budget process of the program s department. Appropriate assessment data support requests by departments for additional funding for changes. The program s assessment data can also be useful in reviewing and revising the college s statements of mission and purpose. Parkland s academic assessment process seeks to improve transfer programs, career programs, and general education offerings. In spring 1998, Parkland s chief academic officer appointed a small committee of faculty to review the published general education objectives and to make them more measurable. The PCA senate approved the revised objectives in November There is no separate department for general education at Parkland; general education courses are offered in five different academic departments. In 1999, on the advice of Peter Ewell, a NCHEMS consultant, the general education academic assessment subcommittee was formed to assess student academic achievement in general education courses. In May 2000, the PCA senate voted to make the curriculum committee responsible for scheduled review and evaluation of the general education objectives, determination of which courses qualify as general education core courses, and review of the general education academic assessment processes and results. The immediate goal of the general education academic assessment subcommittee is to ensure that departmental and cross-disciplinary assessments are regularly designed and conducted. As of May 2002, assessments have been conducted for six general education objectives. The remaining two will be assessed during the academic year, and assessment of all the objectives will be ongoing. Please see Table Table 13.1: ASSESSMENT STATUS OF GENERAL EDUCATION OBJECTIVES The general education objectives in bold have been assessed; the remaining two will be assessed during the academic year. Students will: demonstrate their ability to read, write, listen, and speak effectively; demonstrate their ability to think critically, which includes collecting facts and making decisions based on them, and solving problems, using methods of critical and scientific inquiry; demonstrate their ability to compute and to think and express themselves effectively in quantitative terms; demonstrate their creative potential and their ability to appraise the quality, value, and significance of cultural artifacts, such as literature, sculpture, painting, music, and performing arts; demonstrate their ability to use technology, especially computer technology, to access, retrieve, process, and communicate information; demonstrate their understanding of worldwide political, social, and economic issues, historical and geographical perspectives, the internal and external world, and philosophical ideas; demonstrate their understanding of the necessity of core values in helping them make ethical personal, social, and professional decisions; demonstrate their understanding of the benefits of diversity in cultures, ideas, perspectives, ethnicity, religion, gender, and sexual orientation for a democratic society. Source: General Education Academic Assessment Subcommittee Many necessary processes related to general education are now reviewed annually. Each May, the general education review committee, whose members include the curriculum committee chair, the director of OIRE, the AAC chair, and the general education academic assessment subcommittee chair, will review one of the five general education areas (natural sciences, mathematics, communications, social and behavioral sciences, humanities and fine arts). This five-year cycle is synergistic with Parkland s program review rotation for the ICCB. The purpose of the annual May review is to verify that individual departments general education courses, course information forms, and syllabi appropriately reflect multiple general education objectives. ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT 111

4 In spring 2001, the Chapter 13 committee conducted surveys and interviews to evaluate the effectiveness of the college s academic assessment efforts. Of 196 PAE members who were sent surveys, 96 (49%) responded; of 364 part-time faculty who were sent surveys, 43 (12%) responded. Of the 41 career program directors, department chairs, and assessment coordinators who were sent surveys, 25 (61%) responded. Randomly encountered students in various areas of the college were also surveyed. In addition, the Chapter 13 committee interviewed the AAC chair and the general education academic assessment subcommittee chair. The results of the survey conducted by the Chapter 13 committee are consistent with the faculty/ staff/administrator climate survey results. The Chapter 13 committee s survey questions are available in the resource room; responses are discussed throughout this chapter. Using the North Central Association s Handbook of Accreditation and the Higher Learning Commission s Assessment Matrix, the Chapter 13 committee identified strengths and challenges in Parkland s academic assessment process. For the most part, the implementation of Parkland s assessment program is making progress (level two) or is in maturing stages of continuous improvement (level three). The committee also identified an area that is just beginning implementation (level one). Institutional Culture: Collective and Shared Values Parkland s academic assessment program exhibits mainly a level two pattern of characteristics in this category. The assessment program is a campus-wide network involving academic and student services. Assessment program materials affirm NCA s statement, Assessment is a faculty process, with administrative support, for the benefit of the students. Student academic achievement cannot be fully addressed without the participation of student services staff, librarians, and administrators. The action plan of the academic assessment program states that faculty can also look to the entire college community for solutions for improving learning. Complementing the academic assessment program, the institutional effectiveness program provides ongoing evaluation and improvement in Parkland s support services. Please see Chapter 16, Institutional Effectiveness. Assessment of student learning is valued across the institution, departments, and all programs, evidenced by each department s representation on the AAC and by the college s institutional effectiveness program for academic support services. In their survey responses, 70 full-time faculty (73% of respondents) stated that they realize the value of academic assessment to student learning. For the part-time faculty who have been involved with academic assessment, 28 (66% of respondents) realize its value. Learning lab courses ALR 131 (word attack and basic comprehension I) and ALR 145 (critical thinking I) were developed to help dental hygiene students with additional instruction in the dental vocabulary and terminology used in national board testing. The need for this instruction became evident to program faculty through academic assessment results. Another example is the study-skills enhancement for mathematics students that is provided by the learning lab s ALM 135 and ALM 136 (topics from trigonometry) and sponsored by the Project G.O.A.L.S. program; it was inspired by student assessment and research into the needs of at-risk students. Parkland is beginning to work at level three in many areas. Chapter 13 survey evidence shows that academic assessment is being used to improve teaching and learning, allowing the college to meet the needs of its various stakeholders: students, employers, transfer institutions, and state, local, and federal government. The college clearly has extended academic assessment to distance learning, adult education programming, non-traditional programs, and programs taught off campus. The college is close to level three in that all departments consider assessment an integral part of their functioning. Survey responses indicated that most full-time and some part-time faculty understand academic assessment s usefulness and its process. However, 17 (40%) part-time faculty respondents and 14 (15%) full-time faculty respondents did not understand its usefulness or process. Survey responses show that some faculty believe that assessment is merely add-on work necessary for continued accreditation. The Chapter 13 committee believes all faculty need to be convinced that academic assessment is useful. 112 ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT

5 Institutional Culture: Mission In this category, implementation of Parkland s academic assessment program exhibits progress consistent with level three, maturing stages of continuous improvement. Parkland s statement of mission and purposes reflects the high value the college places on student learning and achievement of academic and career goals, and all of Parkland s academic programs have published statements regarding their purposes and educational goals. Parkland s academic assessment program materials reflect clear connections to and guidance from Parkland s mission and purposes statement as well as the college s central commitment to assessing student learning and to continuous improvement of academic programs. College-wide, continuous assessment of student academic achievement provides evidence that Parkland is fulfilling its mission by keeping its educational promises to the people it serves, the residents of District 505. Parkland s academic assessment program is designed to provide accountability and efficacy data to college administrators and faculty, who use it to keep the college s programs current and useful, allowing students to meet their academic and career goals. Because Parkland s mission and purposes undergird academic assessment, the Chapter 13 committee agrees with the recommendation in Chapters 4 and 16 that the PCA, the institutional effectiveness (IE) committee, and the AAC review these statements every two years. Shared Responsibility: Faculty Parkland has provided many opportunities for all faculty to learn about academic assessment and to become involved in the process. Peter Ewell s September 2001 consulting report stated that there is good evidence in many cases of careful program-faculty reflection about how their assessment methods could be improved, which methods are working well, and which need to be revised or abandoned. Implementation in the shared responsibility: faculty category has matured to level three with the full-time faculty. Each of the 96 full-time faculty survey respondents indicated awareness of academic assessment. All of the departments have developed measurable objectives. Faculty members are adding direct and indirect measures of student learning to their course syllabi and revising academic assessment plans as needed. Only 25 (60%) part-time faculty respondents indicated awareness of academic assessment, a finding corroborated by the spring 2001 faculty/staff/administrator climate survey s discovery that 42% of 21 part-time faculty respondents (9 persons) had no experience with academic assessment. However, since the part-time faculty s only contractual obligations are to teach assigned classes and hold office hours, maximal part-time faculty awareness of and participation in academic assessment should not be expected compared to full-time faculty, whose formal job descriptions include this academic assessment responsibility. When the compliance rate for academic assessment plans fell from 91% in 1999 to 83% in 2000, the AAC discovered that the decrease was due to the curriculum committee s approval of new programs without assessment plans. Now, according to revised curriculum committee policy and procedures, each new program seeking approval must submit an academic assessment plan. The AAC chair and an AAC representative from the requesting department review each new plan before the approval hearing for the new program can proceed. This change resulted in a 99% compliance rate in Parkland s faculty have evolved to the maturing stages of continuous improvement, level three, in making program changes as a result of academic assessment. Many programs have demonstrated effective assessment practices and routinely link assessment results to program improvement. One example pertains to the suggested mathematics course sequence that students had been advised to take. After a review of academic assessment and placement results in MAT 094 (pre-algebra), the placement process into MAT 094 was revised to enhance student learning and success in the course. A mathematics transition course (TRN 072) was created as a result of the collaborative efforts of the Mathematics department, the counseling center, the learning lab, and adult education. All full- and part-time faculty are encouraged to attend campus-wide meetings about academic assessment, and all are invited to attend preparation and development week workshops and seminars. However, the Chapter 13 committee believes that efforts to educate students and part-time faculty about the value of academic assessment must be intensified. ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT 113

6 At level three, faculty are beginning to explore and implement changes for program improvement. Assessment data, analysis, and actions for improvement/changes can spur recommendations for program resource requests. For example, in mass communications, academic assessment data affected planning and resource allocation, directing the faculty to inactivate the visual arts program, revise the graphic design program, and add a new program, Web design. Additional resources were necessary to make changes in the graphic design program and to develop the new program. In the occupational therapy assistant (OTA) program, surveys of employers and graduates identified weaknesses in splint fabrication. The curriculum was revised to increase the number of splinting labs from one to five over a two-year period. Based on academic assessment results, requests for instructional materials and equipment to improve splinting, hand skills, and cognitive assessment were approved by the administration, and the OTA program s budget was increased in fall 1999, when the revisions were implemented. Students splinting skills have since improved. Shared Responsibility: Administration and Board Parkland is at level three in this category. Parkland s academic assessment program enjoys the strong endorsement of college administrators and their commitment of time, human resources, and funds. The EVP, who is the college s chief academic officer, is responsible for overseeing the ongoing operation of the assessment program. He has served as an ex-officio member of the AAC since its inception and provides administrative support and encouragement. He works with the AAC and the IE committee to direct the development and implementation of programs and policies in student academic achievement. The EVP is also responsible for administering and providing adequate institutional support for the programs to assess student academic achievement as well as linking academic assessment and institutional effectiveness with institutional planning and budgeting. Program directors and department chairs support academic assessment. Responding to the survey, one department chair wrote, I believe that assessment has helped faculty examine their own program goals and competencies and arrive at a consensus. Academic assessment was the motivation for the coming together of faculty, and the results benefit the student. Another wrote, I believe there is greater consistency among different sections of the same course. Many program directors have extended the public s knowledge of academic assessment by reporting at their advisory committee meetings the links among academic assessment and program changes, faculty allocation, and resource allocation. Some of the ways the administration supports academic assessment include providing travel money for the North Central annual meeting and other conferences addressing academic assessment and institutional effectiveness, paying an outside rater for English assessment pilot projects, and granting release time to the AAC chair. In 1994 and 1997, the EVP authorized NCHEMS consultant Peter Ewell to guide the faculty in their academic assessment efforts. Peter Ewell reviewed the assessment plans of all programs and course clusters. Parkland s administration and trustees also supported an increase in faculty when academic assessment results showed some areas of concern. Two additional full-time developmental faculty in English and one additional full-time developmental faculty in mathematics were hired in 2001, and a second developmental faculty member in mathematics was hired in A half-time director of developmental education was also hired in Shared Responsibility: Students Parkland needs improvement in this category, which is at level one, the beginning stage of implementation. The college catalog and Web site inform students of Parkland s goals for student learning. The statement of mission and purposes and the general education objectives, printed in the college catalog and class schedule, clearly state the college s educational goals, and there is a statement about college-wide academic assessment in the catalog. However, students appear to know little or nothing about the academic assessment program. The Chapter 13 committee s spring 2001 survey of 39 randomly encountered students revealed that none knew about the college s academic assessment process or program. The AAC has added a student representative to give a valuable viewpoint on academic assessment issues. 114 ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT

7 Institutional Support: Resources Parkland is at level three in this category. There are many resources available to faculty to help with academic assessment, including OIRE, a comprehensive Web site ( index.htm), library reserve holdings about assessment, helpful hint cards to initiate the assessment process, samples of academic assessment plans from many areas of the college, and sample graduate and employer surveys. Additionally, the AAC chair is eager to help individual faculty with their program assessment plans. Since 1993, assessing student academic achievement has been a specific action plan strategy in the college s operational plan, assuring that academic assessment will continue to be sustained and enhanced as necessary. There continues to be a link of assessment outcomes and resource allocation within each department. The program director and department chair surveys indicated that 70% of the college s programs have cited academic assessment results as the reason for a specific request or change. In addition to the learning lab, mathematics, mass communications, and additional-faculty examples mentioned above, department chairs and program directors cited the following ways that academic assessment has affected resource allocation and college planning: Assessment tools have been created to elicit feedback from students in an attempt to collect information on Parkland s success in meeting the academic needs of students. Faculty who teach Psychology 101 want to apply recent innovations involving technology in the classroom. The office careers program was able to buy a color printer because the advisory committee members who evaluated student portfolios suggested that use of color would lend a more professional appearance. Computer aided drafting (CAD) updated their labs with large projection screens for visual demonstrations to better serve the students different learning styles. According to one faculty member, Academic assessment has been our program s road map to improvement over the last five years. Institutional Support: Structures A solid structure has been the foundation for many of Parkland s academic assessment successes; the college has achieved level three implementation status in this category. Faculty receive feedback on their assessment plans from the AAC chair and committee representatives through an annual reporting cycle. The identified changes are included in each program s five-year review process. The effectiveness of changes is monitored yearly; this includes the review of both direct and indirect measures of student learning. As the assessment cycle proceeds, the college s mission and purposes, program goals and objectives, and academic assessment plans will be reviewed and revised based on the data obtained. The reporting format emerged as an area needing improvement. Almost a third of the program directors surveyed were concerned about the difficulty in filling out the assessment reporting form. The AAC has developed a simpler, clearer form to be used in fall 2002 and thereafter. Please see Volume II, Academic Assessment and Institutional Effectiveness Examples. Preparation and development week all-faculty meetings, meetings of the board of trustees, instructional council, department chair council, executive-team meetings, and individual department meetings are all ways that faculty, staff and administrators are becoming knowledgeable about the vocabulary and practices of effective academic assessment. Since fall 1995, the administration has scheduled time during preparation and development week meetings for faculty to work on academic assessment plans and participate in on-campus workshops on academic assessment. At preparation and development week all-faculty meetings, the AAC chair has reported on the college s assessment progress, and faculty from various programs have given presentations on the value of academic assessment to them. Because the AAC is a standing committee of the PCA, the chair submits an annual AAC report to the PCA in December and may give a report at each monthly senate meeting. The AAC chair also reports annually to the board of trustees. ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT 115

8 Summary of Assessment of Student Academic Achievement Strengths The college is committed to the academic assessment process and its value to teaching and learning. A NCHEMS consultant has reviewed the assessment plan for each program or course-cluster and prepared written suggestions for improvement and revision. Parkland has committed the necessary funds for improvement and involvement for all programs and courses. The academic assessment program has become part of the college s daily life. Assessment of Student Academic Achievement Challenge Student awareness and involvement are minimal. Please see Volume II, Academic Assessment and Institutional Effectiveness Examples. Table 13.2: ACADEMIC ASSESSMENT TIME LINE Fall Academic Assessment Committee (AAC) formed. AAC working assumptions developed. Education booklet introducing academic assessment prepared for department chairs Fall Preparation and Development Week: Academic assessment panel discussion Sinclair Community College speakers present Assessment Is like a Box of Chocolates: You Never Know What You Will Get, or What Do Assessment and Apollo 13 Have in Common? Academic assessment workshops for faculty. Academic assessment education sessions for department chairs Assessment of Student Learning at Parkland College educational booklet prepared for entire college. Academic assessment educational sessions for faculty. AAC attends quality-improvement help sessions with Desna Wallin and Jon Ryan, consultants from Iowa. All programs urged to collect some data and begin to analyze them for insights into programs effectiveness. AAC provides new academic assessment guidelines for additional clarification Fall Preparation and Development Week: Faculty receives timetable for completion. All programs to collect and analyze some data during the academic year. Second edition of Assessment of Student Learning at Parkland College distributed. Department chairs submit a status report on academic assessment to the AAC and to Chief Academic Officer. Academic assessment Web page added to Parkland College Home Page. Three AAC committee members attend Academic Assessment conference in Nashville to help the AAC with assessing general education. AAC receives 29 Parkland program assessment plans Spring K. Patricia Cross discusses Classroom Assessment and Research. Barbara Walvoord conducts a Parkland faculty workshop on Primary Trait Analysis. Fall AAC members meet with assessment plan originators to help update the plans. 116 ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT

9 Table13.2: Continued 1998 Spring Peter Ewell, Preparation and Development Week keynote speaker. Curriculum Committee discusses making academic assessment part of new program/course development. General education objectives revision begins. AAC chair attends Summer Assessment Academy in Rapid City, SD. Fall Preparation and Development Week: Jane Hunter, former NCACIHE associate director, presentation to faculty on the assessment process. Revised, measurable general education objectives adopted by the PCA Senate Spring Preparation and Development Week: Center for Excellence in Teaching and Learning presentation on Classroom Assessment Techniques and academic assessment. AAC creates the General Education Academic Assessment Subcommittee. AAC reviews academic assessment plans. Fall Preparation/Development Week: Peter Ewell, keynote speaker, Taking Responsibility for Learning. Mini-courses offered for developing and using assessment tools Spring Preparation/Development Week: Department academic assessment meetings with Peter Ewell for 158 faculty. Joint meeting of Academic Assessment and Curriculum Committees. Fall Class Act, newsletter about academic assessment, published and distributed to faculty Spring Preparation/Development Week: Vincent Tinto presentations on the role of faculty in student retention and strategies for assessing retention and retention programs. Faculty discussions and workshops on assessment development. Focused academic assessment help sessions in the Center for Excellence in Teaching and Learning. Fall Peter Ewell reviews updated plans of program-level assessment Spring Preparation/Development Week: Time available for continuing academic assessment work. Source: Chapter 13 Committee ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT 117

10 118 ASSESSMENT OF STUDENT ACADEMIC ACHIEVEMENT