Brooklyn College Assessment Plan 2014
Brooklyn College Assessment Plan Introduction The principles that underlie the Brooklyn College academic assessment plan include best practices in learning outcomes assessment and program evaluation to support student success and academic program excellence. The College and its various departments are mandated by its own mission and strategic plan, the City University of New York system, and various specialized accreditation bodies, to carry out and document assessment activities. Therefore the plan is guided by and aligned with Middle States Commission on Higher Education Standards, 1 the Brooklyn College Strategic Plan 2011-16, and the City University of New York Performance Management Process. Implementation mechanisms and some specifics will vary depending on the particulars of specific units. General responsibility for implementation belongs to unit/program heads (i.e. Deans, Chairs, Faculty, and Program Directors) and the specifics of implementation will depend on their choices about issues such as resource allocation and assessment priorities. In addition, there will be some variation among the various academic departments in terms of what types of assessment tools are employed depending on the particular field. However, this is in keeping with all of the principles outlined above. The evidence of these activities needs to be shared with the appropriate administrators in order that they may prepare required reports for CUNY, the Middle States Commission on Higher Education and/or other accrediting bodies, and carry on the basic business of the college. In addition, the results of program level assessment activities need to be made available for appropriate use by the college. It is the stated policy of the college, CUNY, and the Middle States Association, as well as a basic principle of good assessment practice that the results of assessment activities not be used for the promotion and tenure decisions of individual faculty, nor used to exclude students from the college or a particular program. 1 Assessing Student Learning and Institutional Effectiveness: Understanding Middle States Expectations (Middle States Commission on Higher Education, 2005). Characteristics of Excellence in Higher Education (Middle States Commission on Higher Education, 2006).See also MSCHE Assessment Task Force Draft Report, 2013 1
Institutional Effectiveness The President s Cabinet will shape overall college policy with respect to the assessment of institutional effectiveness. It will regularly review assessment reports and findings of all academic and administrative units. In order for the College to be successful in serving its students, all units must critically assess their operations to insure the achievement of desired outcomes. Each non-instructional area will articulate its own specific goals and desired outcomes through a process of collective selfexamination and self-reflection. Goals will differ across departments but will all be consistent with the mission of the college and the strategic plan. After goals are established, it is necessary to determine the methods used for measurement. Methods also will vary from unit to unit but may include commercial and locally developed surveys, analysis of institutional data provided by both the BC institutional research office and/or CUNY, focus groups, and criterion based observer ratings. The alignment of categories of outcomes with measures and the mission is illustrated by the chart in appendix 1. 2 Departments will review desired outcomes and assess their performance in light of the available information. Streamlined procedures or new services may result from this process. The assessment effort will inform efforts to improve the functioning of each department and will lead to increased institutional effectiveness. Student Learning Outcomes Assessment Consistent with national best practices in student learning outcomes assessment and the principles outlined above the following will be the primary institutional structures and procedures for learning outcomes assessment. Office of the Director of Academic Assessment The Office of Academic Assessment reports to the Assistant Provost for Planning and Special Projects and coordinates and implements a comprehensive system of assessments of student learning for all graduate and undergraduate programs. It also works with the division of Academic Affairs and its constituent academic departments and administrative units in 2 Hierarchy chart was developed by Michael Anderson from concepts and elements from the following sources: Bennett and Rockwell, 1995, Targeting Outcomes of Programs Mosen Auryan, Using Logic Models in Grant Writing and Program Evaluation CUNY Assessment Council Seminar Presentation, CUNY Graduate Center, October 1, 2010. See also Frechtling. J. (2007) Logic Model Methods in Program Evaluation. 2
achieving and validating excellence through systematic program evaluation, curriculum development, and revision. Having primary responsibility for the college s Outcomes Assessment Program, the office guides and supports departments, faculty, administrators, and staff in the division of Academic Affairs in understanding, articulating, implementing, planning, and coordinating activities required for its outcomes assessment efforts. It coordinates the design of program self-study and review, guides the development of assessment tools, and works with appropriate partners to interpret and analyze data and feedback for purposes of program improvement. The Director works collaboratively with the Office of the Provost, deans and academic department chairpersons, individual faculty, staff, and the Office of Institutional Research to implement the assessment of student learning program. The office also provides direct education about assessment to faculty, staff, and administrators through individual instruction/consulting and a variety of presentations/workshops. Academic Assessment Council This body will consist of faculty representatives from each of the college s five schools appointed by the Provost from a slate of three faculty nominated by the School Deans in consultation with their academic departments. In addition, the Associate Provost for Academic Programs, the Director of Academic Assessment, the Director of Institutional Planning, Research, and Assessment, the Associate Dean of the School of Business and the Assistant Dean of the School of Education will be ex-officio members. The Council will be chaired by a faculty member selected by the body. All members of the Council will be voting members. The Council s responsibilities will include: Reviewing and providing feedback on academic department and interdisciplinary program assessment plans Establishing timetables for regular academic assessment reports Sharing best practices across campus Reviewing the college wide learning goals and forwarding recommendations to appropriate Faculty Council committees. Collaborating with appropriate faculty/academic staff to coordinate and maintain a General Education program assessment system Reviewing and providing feedback on assessment of faculty and student academic support services provided by the Office of the Associate Provost for Academic Programs, the Associate Provost for Faculty and Administration and, eventually, the student services provided by each of the College s five schools and all academic departments. 3
General Procedures The following are the general procedures for developing and maintaining assessment regimens in the various areas. Academic Departments and Interdisciplinary Programs Academic departments and interdisciplinary programs are required to have assessment plans that encompass all individual academic programs. The required elements of these plans are defined in the college s Learning Outcomes Assessment Rubric 3 (Appendix 2). The plans will be reviewed by the Academic Assessment Council and feedback will be provided to departments based on the standards in the rubric. In addition, the rubric will be used to track departmental, program and institutional status in terms of progress toward a comprehensive and well developed system of learning outcomes assessment. As such the departmental assessment plans will be shared not only with the Academic Assessment Council but with the Office of Academic Assessment so that the college can demonstrate its maintenance of a fully developed system for the assessment of student learning to accreditors and other stakeholders. Faculty will determine what they want students to learn - i.e. student learning outcomes - and choose discipline appropriate assessment methods. Departments and their faculties will be required to show how their assessment results are used to inform changes in their programs in order to improve student learning. In addition evidence of assessment activities at the classroom level is seen as good practice in education and involvement in program level activities are seen as valuable service to the college and to one s department. And it is necessary that departments document that assessment results are used to improve student learning and thus maintain and/or strengthen the quality of undergraduate majors and graduate programs. Plans and related documentation will be posted on the website maintained by the Office of Academic Assessment in order to facilitate sharing of the assessment efforts with appropriate stakeholders, including but not limited to accreditation bodies. General Education The Assessment Council will monitor the ongoing assessment of the approved general education curriculum in the same way that it monitors all the assessment of student learning for all academic programs in the college. The Academic Assessment Council, in cooperation with the Associate Provost for Academic Programs, the General Education Program Director, faculty from relevant disciplines and advice from the office of academic assessment, will design and carry out a system of assessment of the achievement of the college wide learning goals. Consistent with national best practices and accreditation standards the system will include 3 We are grateful to Dr. Annita Alting and Dr. Kathy Powell-Manning at City College for allowing Brooklyn College to borrow liberally from their work on this kind of system. 4
multiple measures, both indirect and direct. The system will be directed toward measuring student learning achievement against the college wide learning goals. The general process: 1. Academic Assessment Council will consider and propose to Faculty Council for adoption modifications/revisions of the College wide learning goals. (Appendix 3). 2. The Academic Assessment Council, collaborating with appropriate faculty/academic staff, will create, and/or revise rubrics for the college wide learning goals and any revisions to these goals enacted by Faculty Council. (Faculty development training as needed provided by Office of Academic Assessment) 3. The Assessment Council in collaboration with the appropriate faculty/academic staff will craft a program level assessment plan for general education that includes the elements outlined in the progress rubric. (Appendix 2) 4. The resulting information will be reviewed by the Academic Assessment Council (with analysis support from Office of Academic Assessment and IR Office). Recommendations based on the results will be forwarded to the appropriate bodies for consideration/implementation. Academic Support Areas Within Academic Affairs, a number of specialized services are provided to students and faculty by the offices of the Associate Provost for Academic Programs and the Associate Provost for Faculty and Administration. In these areas, the assessment of a unit s crucial contributions to the success of our students and faculty is more complex. The diversity means that while reporting elements can be somewhat standardized the specifics will vary. However, whatever the exact methods it would be expected that a variety of methods will be used to assess the various aspects of the operation of student and faculty support activities, including measuring outputs (often quantitative), process effectiveness (often engagement and customer surveys), and outcomes (measures of learning or behavior change). 4 Process: 1. Units develop annual goals aligned with the appropriate institutional priorities (i.e. Strategic Plan, PMP indicators, College Wide Learning Goals, etc.) 2. Units indicate measures/benchmarks that indicate performance in the achievement of each goal and methodology for collection of evidence. 3. Units review resulting evidence and respond accordingly. 4 Michael Anderson, Evaluating Academic Support Services Programs CUNY Academic Support Services Forum. CUNY Graduate Center, March 15, 2012. See also the Middle States Commission on Higher Education Assessment Task Force Drat Report (2013) pp.4-5. 5
The Brooklyn College Academic Assessment Council will review unit plans and provide feedback on these activities. The Office of Academic Assessment will provide professional development support where needed and together with the IR office provide technical assistance where needed. Reporting Academic (disciplinary) departments, interdisciplinary programs, and academic support units will report annually on their assessment activities and any changes made as a result. This will facilitate appropriate and timely responses to feedback. In addition, annual reporting will be integrated into existing processes, including department and program annual reports, the CUNY Performance Management Process (PMPs), accreditation, and decennial reviews. Academic departments and interdisciplinary programs, unless otherwise required by specialized accreditation bodies, should use the form in appendix 4. A copy of a completed form for each program in the department should be included in the department s end of year report. In addition, copies should be sent electronically to the Office of Academic Assessment. Supporting documentation (i.e. minutes of department meetings, records of department actions, rubrics, etc.) should be kept on file in the department. For support units, annual assessment activities will be part of annual reporting by the unit head to the appropriate administrative officer (i.e. Vice-Presidents, Associate Provosts, etc.) who will provide a consolidated report to the Academic Assessment Council for review. Unless otherwise required, units should report using the form in appendix 5. The unit will maintain copies of the reports and supporting documentation. These will be forwarded as needed to the appropriate administrative offices to assist in preparing external and internal reports, including the PMPs. 6
Appendix 1 BC Hierarchy of Institutional Effectiveness Assessment 7
B.C. Hierarchy of Institutional Effectiveness Assessment Positive Impact on Students, Community and World Learning OUTCOME Actions OUTCOME OUTCOME Fulfillment of Mission Changes in behaviors and practices Changes in knowledge, attitudes, skills, aspirations National ranking, PMP process, Alumni surveys, CLA, MSCHE and specialized accreditation, Decennial (outside) Program review etc. Assessed by observation and/or survey (i.e. NSSE), Persistence and graduation rates, PMP Data Learning Outcomes Assessment of Programs and General Education. Also CLA, NSSE, and PMP Data Reactions OUTPUT Participation OUTPUT Satisfaction with programs; level of interest; feelings toward activities, educational methods Number and characteristics of people reached; frequency and intensity of contact Local satisfaction Surveys, CUNY SES, Noel Levitz, BC Feedback surveys, Decennial program reviews. Institutional Data. (admissions and enrollment data, program counts etc.)
Appendix 2 Progress Rubric 8
Learning Outcomes Assessment Status Rubric Introduction The learning outcomes assessment status rubric is a tool designed to help gage the status and quality of student learning assessment in the college s programs. Based on Middle States standards and adapted from work begun at City College and around the country, the rubric allows for programs to evaluate themselves in terms of their progress (and the college s) towards a fully developed culture of assessment and evidence based on national best practices. Currently in draft form it will be adjusted in response to experience with its use. The rubric focuses on five elements of a fully developed learning assessment program Assessment Plans, Learning Goals And/Or Outcomes, Syllabi, Assessment Tools, and Use of Assessment Results and asks the program to score itself on a scale of 1-4 for each of the five elements. Programs are responsible for maintaining and reporting evidence that documents their responses to the rubric.
Learning Outcomes Assessment Status Rubric (Draft) Program Element Assessment Plans Learning Goals and/or Outcomes Syllabi Assessment Tools Use of Assessment Results
Example: Program Element Assessment Plans Learning Goals and/or Outcomes Syllabi Assessment Tools Use of Assessment Results MA in X 2 3 2 2 2 BA in Y 3 2 3 2 1 1 = Initial / Needs Work 2 = In Progress / Emerging 3 = Developed 4 = Highly Developed / Good Practice
RUBRIC Rubric Legend 1 = Initial / Needs Work 2 = In Progress / Emerging 3 = Developed 4 = Highly Developed / Good Practice Definition Assessment Plans An assessment plan describes the process to be used to collect evidence on student learning and the use of this information to improve learning. At a minimum, the plan should include: (a) statements of intended student learning, (b) map showing how outcomes align with educational experiences (c) measures of assessment (e.g. tools, rubrics), (d) evidence collection and analysis processes, and (e) use of assessment results in curricular review and improvement, including occasional review of learning outcomes and the assessment process itself. 1 No plan for assessment, or only ad hoc assessment activities 2 Planning on a short-term basis, and/or not faculty-driven or faculty-approved 3 Faculty-approved multi-year plans, cycling though all program outcomes on a 3 to 5 year schedule, including periodic review of the assessment process itself. Instead of program outcomes, the plans may also be organized around resolving problem areas (e.g., learning outcomes assessment to improve retention in early Math courses), content areas (e.g., specializations within a program), or other concept that makes sense for a particular unit / program. 4 Same as previous, plus integration with resource allocation / strategic planning / external reviews
Learning Outcomes Program Learning Outcomes describe the essential knowledge, skills and dispositions required by graduates of a program; Course Learning Outcomes reflect what the faculty in an academic unit collectively identify as the essential knowledge, skills and dispositions to be acquired by students at the end of the course, in alignment with the relevant program learning outcomes. Learning outcomes are stated in terms of observable and measurable student behavior, e.g., The student demonstrates effective communication skills. (Program level); Students will be able to explain orally and in writing what the output from standard analyses in SPSS shows. (Course level). 1 There may be broadly formulated program objectives and goals and/or a mission, but learning outcomes in terms of students knowledge, skills and dispositions are not (consistently) formulated and it is left to individual faculty to formulate learning outcomes for their courses, if any. 2 Learning outcomes on the departmental level are defined, but not (all) in measurable and observable terms, and they may be assigned to courses in a grid, but without considerations of consistency, balance, students developmental level, and/or alignment (streamlining & alignment) 3 (Most) learning outcomes are clearly formulated in measurable and observable terms, assigned to specific courses, and reviewed for internal consistency, balance, developmental level and vertical alignment 4 As 3, plus standards for knowledge, skills and dispositions at the Gen Ed, Bachelor s, or Master s level are defined, where applicable with disciplinary and/or professional standards in mind Definition
Syllabi A syllabus is an outline of a course, minimally describing (a) the course goals in terms of learning outcomes, (b) assignments and instructional strategies to reflect and help achieve the course goals, and (c) the ways in which students are to demonstrate their learning during and after completion of the course. The syllabus also provides (d) practical information such as course description and prerequisites, schedule, instructor s contact information and office hours, location, relevant policies, etc., 1 The syllabus contains no, or very little, information about what students are expected to have learned as a result of the course (a), and also lacks clear information on many of the other elements (b) to (d). E.g., just a list of topics or a short course description. 2 There are course goals or objectives, but they are often not formulated in terms of learning outcomes, and elements (b) to (d) are addressed to some extent. 3 Course goals are formulated in terms of learning outcomes and elements (b) to (d) are addressed fairly completely. 4 Course goals are formulated correctly in terms of student-centered, observable and measurable learning outcomes, elements (b) to (d) are addressed fairly completely, and syllabi are easily accessible for students and other stakeholders. Definition
Assessment Tools An assessment tool is an instrument (survey, rubric, test, evaluation form, etc.,) that has been designed to collect reasonably accurate and truthful data about the extent to which a course or program succeeds in achieving each specific learning outcome across students in the course or program. Direct assessment tools evaluate actual student work and behaviors related to specific learning outcomes, and indirect assessment tools evaluate students and/or others perceptions of the achievement of relevant learning outcomes, and / or satisfaction with instruction, advisement, support services, etc., 1 Only grades and grade distributions are used to assess student learning 2 In addition to grades, indirect measures (e.g., student satisfaction surveys) are used 3 Tools for direct assessment of most key learning outcomes are available, but they may have some shortcomings in terms of validity, reliability, fairness, and usefulness, and/or be overly labor-intensive or user-unfriendly 4 There are at least two tools (one direct, one indirect) for assessing each key learning outcome relevant to a course / program, and these tools evaluate student learning efficiently, in a valid, reliable and fair manner, providing useful information for decision-making and to determine if standards have been met and/or interventions worked as intended. Definition
Definition Use of Assessment Results ( Closing the Loop ) This step involves an instructor s or unit s interpretation (analysis) of the information from data summaries, and making recommendations, supported by the data, for course and program changes that will improve student learning. The use of assessment results also involves summarizing, reporting and publishing the findings and recommendations for internal and external purposes. 1 Assessment results, if any, do not play a role in curricular decision-making, resource allocation and improvement efforts, or are used selectively, e.g., only when they confirm desired outcomes and/or help make a case for desired resources, and/or are used punitively to deny resources, promotion or tenure, or otherwise inappropriately 2 (Some) Individual faculty use assessment results to improve (student learning in) their own courses 3 Assessment results are used most of the time to guide course and curriculum changes and to measure if changes have the desired effect 4 As 3, and results are also used systematically in resource allocation and curricular planning, and relevant results are reported in an accessible manner for accountability purposes and shared with stakeholders
Appendix 3 BC Learning Goals In order to prepare its graduates to be informed and responsible citizens of the world; and to provide them with a foundation for life-long learning and the potential for leadership: Brooklyn College Expects That All of its Graduates Will 1. Be able to think critically and creatively, to reason logically, to reason quantitatively, and to express their thoughts orally and in writing with clarity and precision; 2. Be able to make sound moral and ethical judgments; 3. Understand the arts, histories and cultures of the past as a foundation for those of the present; 4. Understand the development and workings of modern societies in an interdependent world; 5. Acquire the tools that are required to understand and respect the natural universe; 6. Understand what knowledge is and how it is acquired by the use of differing methods in different disciplines; and be able to integrate knowledge from diverse sources. 7. Understand the necessity for tolerance and appreciate individual and social diversity. 9
Appendix 4 Academic Program Reporting Form 10
Degree Program: Department: College: ASSESSMENT REPORT (Year) Program Coordinator: E-mail: Phone: Departmental/Program Mission Statement Program Learning Goals. Student Learning Outcome Assessment Method Criterion/Benchmark Schedule/ Frequency Results of Assessment/ Interpretation of Findings Use of Results/Action Plan What will students know, be able to do, or value? Should be Specific, Measurable, and aligned to a program learning goal How will the outcome be measured? Who will be assessed, when, and how often? How well should students be able to do on the assessment? Expected level of student performance How often will the information be collected and analyzed? What does the data show? How were/will results used/be used for improvement? Goal #......
Appendix 5 Academic Support Unit Reporting Form 11
Academic Support Unit Reporting Form COLLEGE-WIDE PRIORITIES UNIT GOALS PROGRAM ACTIVITIES AND OUTCOMES PMP goal/target Strategic Plan Theme/Action Item MSCHE Self- Study Recommendation Unit Goal Action/Activity Anticipated outcomes and measure/benchmark. (If any)