TO: Chief Academic Officers FR: Central Office of Academic Affairs RE: Pathways Year-Three Review DT: September 15, 2017

Similar documents
The City University of New York

Implementing Our Revised General Education Program

STUDENT LEARNING ASSESSMENT REPORT

Evaluation of a College Freshman Diversity Research Program

Math Pathways Task Force Recommendations February Background

Mathematics Program Assessment Plan

Early Warning System Implementation Guide

The Teaching and Learning Center

SECTION I: Strategic Planning Background and Approach

Data Glossary. Summa Cum Laude: the top 2% of each college's distribution of cumulative GPAs for the graduating cohort. Academic Honors (Latin Honors)

CONNECTICUT GUIDELINES FOR EDUCATOR EVALUATION. Connecticut State Department of Education

BARUCH COLLEGE THE CITY UNIVERSITY OF NEW YORK

Individual Interdisciplinary Doctoral Program Faculty/Student HANDBOOK

McNeese State University University of Louisiana System. GRAD Act Annual Report FY

Millersville University Degree Works Training User Guide

Program Change Proposal:

U VA THE CHANGING FACE OF UVA STUDENTS: SSESSMENT. About The Study

SUNY Downstate Medical Center Brooklyn, NY

TABLE OF CONTENTS Credit for Prior Learning... 74

OFFICE OF ENROLLMENT MANAGEMENT. Annual Report

Developing an Assessment Plan to Learn About Student Learning

Handbook for Graduate Students in TESL and Applied Linguistics Programs

Undergraduate Program Guide. Bachelor of Science. Computer Science DEPARTMENT OF COMPUTER SCIENCE and ENGINEERING

National Survey of Student Engagement (NSSE) Temple University 2016 Results

What Is The National Survey Of Student Engagement (NSSE)?

10/6/2017 UNDERGRADUATE SUCCESS SCHOLARS PROGRAM. Founded in 1969 as a graduate institution.

GRADUATE PROGRAM IN ENGLISH

Core Strategy #1: Prepare professionals for a technology-based, multicultural, complex world

NCEO Technical Report 27

SURVEY RESEARCH POLICY TABLE OF CONTENTS STATEMENT OF POLICY REASON FOR THIS POLICY

learning collegiate assessment]

College of Liberal Arts (CLA)

Bellevue University Bellevue, NE

Curricular Reviews: Harvard, Yale & Princeton. DUE Meeting

BENCHMARK TREND COMPARISON REPORT:

M.S. in Environmental Science Graduate Program Handbook. Department of Biology, Geology, and Environmental Science

University of New Hampshire Policies and Procedures for Student Evaluation of Teaching (2016) Academic Affairs Thompson Hall

Graduate Division Annual Report Key Findings

Curriculum Development Manual: Academic Disciplines

EVALUATION PLAN

Definitions for KRS to Committee for Mathematics Achievement -- Membership, purposes, organization, staffing, and duties

Undergraduates Views of K-12 Teaching as a Career Choice

Undergraduate Admissions Standards for the Massachusetts State University System and the University of Massachusetts. Reference Guide April 2016

San Diego State University Division of Undergraduate Studies Sustainability Center Sustainability Center Assistant Position Description

STUDENT ASSESSMENT AND EVALUATION POLICY

Spring Valley Academy Credit Flexibility Plan (CFP) Overview

Academic Advising Manual

STEM Academy Workshops Evaluation

EDUCATIONAL ATTAINMENT

TULSA COMMUNITY COLLEGE

2015 High School Results: Summary Data (Part I)

Mapping the Assets of Your Community:

P920 Higher Nationals Recognition of Prior Learning

University of Maine at Augusta Augusta, ME

READY OR NOT? CALIFORNIA'S EARLY ASSESSMENT PROGRAM AND THE TRANSITION TO COLLEGE

ECON 365 fall papers GEOS 330Z fall papers HUMN 300Z fall papers PHIL 370 fall papers

Meta-Majors at Mott Community College

National Collegiate Retention and Persistence to Degree Rates

Facts and Figures Office of Institutional Research and Planning

Assessment Report Univ. North Carolina Asheville SA - Dean of Students

ADMISSION TO THE UNIVERSITY

ACADEMIC AFFAIRS GUIDELINES

Writing a Basic Assessment Report. CUNY Office of Undergraduate Studies

JOB OUTLOOK 2018 NOVEMBER 2017 FREE TO NACE MEMBERS $52.00 NONMEMBER PRICE NATIONAL ASSOCIATION OF COLLEGES AND EMPLOYERS

Review Panel Report Oregon State University. Science and Mathematics Education Graduate Program

Majors, Minors & Certificate Programs 34 The Majors 35 The Minors & Certificate Programs 36

2010 ANNUAL ASSESSMENT REPORT

Credit Flexibility Plan (CFP) Information and Guidelines

Records and Information Management Spring Semester 2016

SORRELL COLLEGE OF BUSINESS

Audit Of Teaching Assignments. An Integrated Analysis of Teacher Educational Background and Courses Taught October 2007

HIGHLAND HIGH SCHOOL CREDIT FLEXIBILITY PLAN

St. John Fisher College Rochester, NY

Policy for Hiring, Evaluation, and Promotion of Full-time, Ranked, Non-Regular Faculty Department of Philosophy

EXECUTIVE SUMMARY. Online courses for credit recovery in high schools: Effectiveness and promising practices. April 2017

PATTERNS OF ADMINISTRATION DEPARTMENT OF BIOMEDICAL EDUCATION & ANATOMY THE OHIO STATE UNIVERSITY

Transportation Equity Analysis

GUIDE TO EVALUATING DISTANCE EDUCATION AND CORRESPONDENCE EDUCATION

DEPARTMENT OF POLITICAL SCIENCE AND INTERNATIONAL RELATIONS. GRADUATE HANDBOOK And PROGRAM POLICY STATEMENT

2005 National Survey of Student Engagement: Freshman and Senior Students at. St. Cloud State University. Preliminary Report.

Robert S. Unnasch, Ph.D.

Validation Requirements and Error Codes for Submitting Common Completion Metrics

Number of students enrolled in the program in Fall, 2011: 20. Faculty member completing template: Molly Dugan (Date: 1/26/2012)

University of Toronto

1.0 INTRODUCTION. The purpose of the Florida school district performance review is to identify ways that a designated school district can:

Hanover College confers the Bachelor of Arts degree when the following conditions have been met:

Executive Summary. Laurel County School District. Dr. Doug Bennett, Superintendent 718 N Main St London, KY

2012 ACT RESULTS BACKGROUND

ACADEMIC ALIGNMENT. Ongoing - Revised

Guidelines for the Use of the Continuing Education Unit (CEU)

University of Toronto

National Survey of Student Engagement Executive Snapshot 2010

Strategic Planning for Retaining Women in Undergraduate Computing

Timeline. Recommendations

The University of North Carolina Strategic Plan Online Survey and Public Forums Executive Summary

Program Information on the Graduate Certificate in Alcohol and Drug Abuse Studies (CADAS)

THEORY/COMPOSITION AREA HANDBOOK 2010

Differential Tuition Budget Proposal FY

8. Prerequisites, corequisites (If applicable) Prerequisites: ACCTG 1 (Financial Accounting) ACCTG 168 (Tax Accounting)

Enrollment Trends. Past, Present, and. Future. Presentation Topics. NCCC enrollment down from peak levels

Transcription:

Office of Academic Affairs TO: Chief Academic Officers FR: Central Office of Academic Affairs RE: Pathways Year-Three Review DT: September 15, 2017 The June 2011 Board of Trustees resolution on Pathways stipulated that Pathways policies and processes be reviewed and evaluated each year for three years beginning in 2013, and every three years thereafter, to modify them as necessary to improve them or to meet changing needs. In September 2016 the Central Office of Academic Affairs released the Pathways Year-Two review that summarized findings of an evaluation of available data after completion of the second full year of Pathways implementation. At that time, and at subsequent meetings, the year-two faculty review committee presented the Central Office of CUNY with a number of questions and requested data on various aspects of Pathways. The data provided in response to these queries formed the content of the second-year review and was used as a guide to the structure and content of the year-three review. Attached is the Pathways Year-Three Review. The Central Office of Academic Affairs compiled this report after the completion of the third year of implementation with contributions from the year-two faculty review team, the chairs and faculty members of the Common Core Course Review Committee, the college Pathways appeals officers, and the student focus group participants. We will make the Pathways Year-Three Review accessible to the CUNY community via the OAA Pathways web site. The Central Office of Academic Affairs will continue to monitor, track, and make public Pathways data on an annual basis. While the next mandated review is scheduled for 2019-2020, efforts are underway to explore options for a comprehensive external review and analysis of Pathways implementation and effectiveness. 0

Pathways General Education Initiative Year-Three Review City University of New York Office of Academic Affairs September 2017 1

TABLE OF CONTENTS Background and Rationale... 3 Table 1: Number and Percent of First-Time Freshmen and Transfer Students Enrolled in CUNY Baccalaureate Programs... 3 Overview of the Pathways Initiative... 4 Table 2: Pathways General Education Common Core Requirement... 5 Implementation... 6 Seamless Transfer... 6 Common Core Course Review Committee... 6 Approved Pathways Common Core Courses... 7 Table 3: Number of Pathways Common Core Courses by Area (as of December 2015)... 8 Undergraduates Enrolled in Pathways... 8 Figure 1: Percent of Degree-Seeking Undergraduates Enrolled in Pathways, Fall 2015... 8 Student Knowledge about Pathways... 8 Figure 2: Students Self-Reported Familiarity with General Education Requirement... 9 Figure 3: Students Self-Reported Familiarity with Course Transfer Guarantees... 9 Major Gateway Course Transfer... 10 Table 4: Transfer Status of Incoming Major Gateway Courses at Receiving Colleges, Spring 2016... 11 Table 5: Transfer Status of Incoming Major Gateway Courses at Receiving Colleges, by Major... 12 Student Appeals Process Review... 12 Early Outcomes of the Pathways Initiative... 15 Efficiency of Credit Transfer... 15 Table 6: Credits Transferred for and Accumulated by Transfers Within CUNY... 15 GPA and Credit Attainment after One Year... 16 Table 7: GPA and Credit Accumulation of First-Time Freshmen and Transfers Within CUNY... 16 Retention Rates and Transfer Type... 17 Table 8: One-Year Retention Rates of First-Time Freshmen... 17 Table 9: Trends in Transfers into CUNY Baccalaureate Programs... 17 Course-Taking Patterns by Discipline... 17 Table 10: Trends in First-Time Freshman Course-Taking, by Discipline... 18 Table 11: Trends in Undergraduate Course-Taking, by Discipline... 19 Next Steps... 20 2

Background and Rationale The Pathways Initiative was designed to strengthen curricular alignment across CUNY s undergraduate colleges and improve the efficiency of credit transfer throughout the system to ensure that all students can complete a meaningful and rigorous program of study in a reasonable and timely manner. The Pathways Initiative aims to: enhance transfer students progress toward degree completion; maintain CUNY s commitment to the highest academic standards and the faculty s special responsibility for courses and curriculum; and provide colleges with the flexibility to maintain their distinctive identities and traditions. June 27, 2011 Board of Trustees Resolution on Creating and Efficient Transfer System The CUNY-wide common curricular structure created through Pathways allows the smooth and efficient transfer of students credits from any CUNY college to all other CUNY colleges. Prior to its implementation in Fall 2013, students who transferred would often find that some course credits earned at one college did not match course requirements at another college; as a result, the credits were not applied to degree requirements. In some cases, credits were applied as elective credits, which may have done little to help students advance toward degree completion. In other cases, CUNY colleges rejected course credits outright, awarding zero credits for completed coursework. The increase in both the number and proportion of transfer students entering CUNY contributed to the need to create a University-wide common curriculum structure. As shown in Table 1, since 2009, the majority of students enroll in a baccalaureate program at CUNY having transferred from another college, while only about one-third of all students enter as a first-time freshman. The Pathways Initiative was CUNY s response to serve its students better, especially transfers and future transfers. Table 1 Number and Percent of First-Time Freshmen and Transfer Students Enrolled in CUNY Baccalaureate Programs 2009-2010 2010-2011 Academic Year 2011-2012 2012-2013 2013-2014 2014-2015 Number of First-Time Freshmen 13,289 12,798 12,761 12,620 12,763 13,141 % of First-Time Freshmen 37.2 36.1 34.3 33.8 34.6 32.9 Number of Transfer Students 22,478 22,641 24,465 24,701 24,158 26,775 % of Transfer Students 62.8 63.9 65.7 66.2 65.4 67.1 Total New Students 35,767 35,439 37,226 37,321 36,921 39,916 Data Source: Institutional Research Database (IRDB) 3

Overview of the Pathways Initiative Following the 2011 Board of Trustees Resolution, or the Pathways Resolution, the Pathways Initiative was to be fully implemented beginning in Fall 2013 for all CUNY first-time freshmen and transfer students. Students who entered CUNY prior to Fall 2013 could opt in to the new curricular requirements. The Pathways Resolution provides transfer credit guarantees for all three components of the curriculum general education, majors, and electives. These guarantees apply for all students, regardless of the type of degree program they are transferring from or into, or whether they have earned a degree or not prior to transfer. 1. The Resolution specifies that general education at CUNY consist of a 30-credit Common Core requirement for all colleges across the University, with up to 12 additional credits, 1 referred to as the College Option, required for students enrolled in baccalaureate programs. A University Task Force proposed a flexible, yet uniform, curricular structure for general education and the early part of the major that would allow for seamless transfer. A broad curricular framework would provide the flexibility needed for CUNY colleges to design their own general education program and maintain many of their existing requirements and course offerings. The Task Force developed a set of broad Learning Goals for each distribution area that were modeled after the Essential Learning Goals developed by the Association of American Colleges and Universities (AACU) Liberal Education and America s Progress (LEAP) initiative. This common structure reduced the number of required general education credits at most CUNY colleges. 2 The 30-credit Pathways Common Core framework stipulates credit distribution by area, as shown in Table 2. While specific learning outcomes are associated with each area, a wide variety of courses and disciplines are included. The Pathways transfer guarantees ensure that once a student successfully completes a course in a particular Common Core area, the credits transfer to and satisfy the requirements for that Common Core area at any other CUNY college. 3 Requirements for the College Option vary across colleges, and in some cases, vary across different schools within the same college. 1 http://www2.cuny.edu/about/administration/offices/undergraduate-studies/pathways/gened/college-option/ 2 Prior to Pathways, on average, CUNY colleges required 52 general education credits, with some schools requiring as many as 63 credits. 3 Further detail about the Common Core areas, including the learning outcomes associated with each, can be found in the CUNY Pathways Task Force s Final Recommendation to the Chancellor (http://www1.cuny.edu/mu/academic-news/files/2011/12/commoncorestructurefinalrec.pdf). 4

Table 2 Pathways General Education Common Core Requirement General Education Number of Credits Areas of Required Common Core 12 4 English Composition 6 2 Mathematical and Quantitative Reasoning 3 1 Life and Physical Sciences 3 1 Areas of Flexible Common Core 18 6 World Cultures and Global Issues 3 1 U.S. Experience in Its Diversity 3 1 Creative Expression 3 1 Individual and Society 3 1 Scientific World 3 1 One additional course from any one of the five areas above 3 1 Number of Courses 2. Pathways provides transfer guarantees for gateway courses in selected majors with high numbers of transfers. As of June 2016, those majors are: Biology; Business; Criminal Justice; Economics; English; Nursing; Political Science; Psychology; Sociology; and Teacher Education. Each college identified between three and six Major Gateway courses in each of the selected majors that should be accepted as entry-level courses for beginning the major at any CUNY college offering the major. 3. Pathways guarantees that all courses completed for credit at one CUNY college be accepted for credit upon transfer to any other CUNY college, with the understanding that courses that do not meet the Common Core or major requirements may count for elective credit. 5

Implementation Seamless Transfer Before Pathways, thousands of courses from CUNY colleges were not accepted for credit at other CUNY colleges. That is, students who had completed such coursework did not receive elective credit or any other type of credit upon transfer. In Fall 2013, the University Registrar s Office collaborated with each campus Registrar to recode such courses so that these courses now at least receive elective credit upon transfer. In addition to ensuring that credits earned at one CUNY college transferred for credit to any other CUNY college, Pathways addressed the issue of credit transfer toward degree requirements. Before Pathways, courses transferred toward degree requirements if they were coded as equivalent in the transfer credit rules within CUNYfirst, the University s student information system. For example, if College A required Introductory Biology for the Biology major, a student transferring from College B must have completed a course deemed equivalent to College A s Introductory Biology course to receive credit toward the requirement. With Pathways, Common Core and College Option courses contain Requirement Designation codes in CUNYfirst that transcend course equivalencies and result in the automated application of these courses toward Common Core or College Option requirements upon transfer. The coding of Pathways courses with these Requirement Designation codes has resulted in the seamless transfer of thousands of courses. Common Core Course Review Committee The 2011 Pathways Resolution requires faculty to review all courses to be included in the Common Core curriculum to ensure that all learning outcomes are satisfied. The Common Core Course Review Committee (CCCRC) was established in 2012 to meet this requirement. During the early implementation phase, there were eight CCCRC subcommittees, one for each of the eight areas of the Common Core. When the volume of courses decreased, three subcommittees were able to cover the Common Core areas. By the fourth year, two subcommittees reviewed course submissions in all eight areas. Each subcommittee is composed of faculty members from the CUNY colleges that offer undergraduate degrees. Each subcommittee is led by a chair, who is a faculty member in that subcommittee. Faculty members are compensated for their work and serve a 1-2 year term. 4 After CCCRC approval, courses are submitted to the Chancellor s University Report before being coded as Common Core courses in CUNYfirst and Degree Works, the degree audit system. 4 Current committee members and college affiliation are posted on the Pathway CCCRC web page. http://www2.cuny.edu/about/administration/offices/undergraduate-studies/pathways/about/ccrc/ 6

Colleges fill out Course Submission Forms to explain the course assignments and activities that address learning outcomes. Colleges also submit a sample syllabus for each course. The CCCRC considers both the Course Submission Form and the syllabus when determining if a course meets the learning outcomes. The syllabus should provide additional context, but does not have to repeat all of the same information included on the Course Submission Form, as long as the two documents do not conflict. Also, because different course sections will be taught by different professors using different approaches, the syllabus submitted as part of the CCCRC review process is considered representative. As part of this third-year review process, the CCCRC chairs worked with their subcommittees to review and assess their processes. Together, the three chairs developed a set of guiding questions for their subcommittees, and then each subcommittee wrote a response with recommendations for improvements. While the CCCRC agreed that the review process resulted in greater confidence in the overall quality of the curriculum at the colleges, the subcommittees provided the following recommendations for improving the review process: There needs to be better communication between the course review subcommittees and the faculty who are preparing courses and syllabi for review. While each CUNY campus has a liaison who uploads course proposals and communicates committee votes back to the campus, there have been calls for increased transparency about the process and additional clarification about expectations. The subcommittees could post on the Pathways website a written set of criteria and guidelines that detail how each course submission is evaluated according to the learning outcomes. The subcommittees could be regularly reminded of the scope of their task: determining whether a given course satisfies the Pathways learning outcomes. Too often, subcommittees evaluate courses based on pedagogy, content, mode of instruction, and other factors that are not within the purview of Pathways. An orientation session for faculty new to serving on the CCCRC is recommended. A written manual would be helpful as well. Guides could be developed to help faculty at the colleges better understand what information needs to be submitted. Approved Pathways Common Core Courses As of Fall 2016, there are over 2,300 approved Pathways Common Core courses offered across the University, as shown in Table 3. With the exception of the STEM variant courses, these courses have been approved by the CCCRC. STEM variant courses are not required to be reviewed by the CCCRC because these courses are required for at least one major at the college and were not specifically designed to address general education learning outcomes. Although Pathways requires that most Common Core courses consist of 3 credits, STEM variant courses may be offered with more than 3 credits in the areas of Mathematical and Quantitative Reasoning, Life and Physical Science, and Scientific World, as long as a sufficient number of 3-credit options are also available to students in these areas. STEM variant courses allow students who enter CUNY with advanced math or science skills, as well as 7

those in STEM majors with particular math or science course sequencing requirements, to take these more challenging courses within the general education curriculum. A college may offer a STEM variant course in more than one Pathways area. Table 3 Number of Pathways Common Core Courses by Area (as of December 2016) Number of Courses Areas of Common Core All (STEM Variant) English Composition 77 Mathematical and Quantitative Reasoning (MQR) 176 (133) Life and Physical Sciences 116 (38) World Cultures and Global Issues 586 U.S. Experience in Its Diversity 223 Creative Expression 285 Individual and Society 335 Scientific World 269 (93) Life and Physical Sciences or Scientific World 1 233 (233) MQR or Scientific World 1 12 (12) Total 2312 (509) 1 These courses can fulfill the requirements of either of the two areas. Undergraduates Enrolled in Pathways As in Figure 1, by Fall 2015, the majority of CUNY students 81 percent of all degree-seeking students were following the Pathways curriculum, as would be expected. Figure 1 Percent of Degree-Seeking Undergraduates Enrolled in Pathways, Fall 2015 Undergraduate Total Baccalaureate Program Associate Program 80.6 74.5 87.6 Data Source: CBIL (CUNY Business Intelligence Light) Database. CBIL reflects daily refreshed enrollment, based on day-old copies of CUNYfirst tables in the Administrative Data Warehouse (ADW). The enrollment figures from CBIL do not match standard enrollment reports from OIRA. Student Knowledge about Pathways 0 10 20 30 40 50 60 70 80 90 100 Figures 2 and 3 show students responses to questions about familiarity with the general education requirements and course transfer guarantees provided by CUNY s Pathways Initiative. These responses were collected in Spring 2015, more than three semesters after Pathways was first launched. An % 8

independent consultant led the focus group sessions. There were eight focus groups consisting of a total of 57 students from four CUNY colleges: two community and two senior colleges. The focus groups provided a first glimpse of student impressions and understanding of Pathways and general education. While not intended to be exhaustive or conclusive, the focus groups shed some first light on student understanding not only of Pathways and general education, but of their understanding of the curriculum in general and their experience with college life. Although responses varied across colleges and for different student groups, more than half of all students, 56 percent, indicated they are Very familiar with their general education requirements, as Figure 2 depicts. However, Figure 3 shows less than one-tenth of the students, 7 percent, said they are Very familiar with the course transfer guarantees, and nearly one-third, 30 percent, answered Don t know anything about the transfer guarantees. Students responses during the focus group sessions indicated knowledge of the Major Gateway courses might be especially limited. However, the majority of focus group participants had majors that do not participate in the Pathways Major Gateway plan. Figure 2 Students Self-Reported Familiarity with General Education Requirement "Are you familiar with your general education requirement?" 100% 80% 60% 40% 20% 0% 56% 44% Overall Sample (n = 57) 77% 23% 28% 72% 33% 67% 50% 50% 0% 0% 0% 0% 0% FTF Community College (n = 30) Transfer Students (n = 18) FTF Senior College (n = 3) Opt-in Students (n = 6) Very Familiar Know a little Don't know anything Figure 3 Students Self-Reported Familiarity with Course Transfer Guarantees 100% "Are you familiar with the course transfer guarantees provided by CUNY's Pathways initiative?" 80% 60% 40% 20% 0% 7% 63% 60% 30% Overall Sample (n = 57) 20% FTF Community College (n = 30) 6% 72% 20% 22% Transfer Students (n = 18) 33% 67% 0% 0% FTF Senior College (n = 3) 50% 50% Opt-in Students (n = 6) Very Familiar Know a little Don't know anything 9

Analysis of the eight focus groups identified several major themes: 1. Most students, regardless of student type or college type, have a neutral to positive opinion of the general education requirements. Students comments included: The general education requirements provide me with exposure to a broad range of learning topics and (As a first-time freshman in 2013) I accept the general education program as what is required of me as a college student. 2. A minority of students hold a neutral to negative opinion of the general education requirements. Some students commented: There are too many requirements and I would like to focus more on my major. 3. Community college students overwhelmingly hold the opinion that the transfer guarantee policies will be helpful to them and other CUNY students transferring between CUNY colleges. Students comments included: This will save me time and money when I transfer to a senior college and I wish these guarantees extended to non-cuny schools. 4. Transfer students had mixed experiences with their transfer credits. Some students commented: The transfer guarantees were helpful to me when I transferred to a CUNY senior college and The transfer guarantees are a good idea but did not work as easily for me as I expected. 5. Most students do not have an understanding of the College Option component of Pathways. 6. Most students believe that the Major Gateway component of Pathways will be helpful to CUNY students; many students are unfamiliar with this policy. 7. Perception of advisement varies widely within and among institutions. As a result, some students felt they received proper advisement about Pathways from their advisors, while others indicated that they self-educated using Degree Works and their college websites. Major Gateway Course Transfer As required by Pathways, faculty committees designated a minimum of three gateway courses leading into several of CUNY s most popular majors. Students who anticipate pursuing one of these majors can take the designated courses and transfer them for major credit seamlessly between CUNY colleges that offer the major. An analysis has been conducted to evaluate whether this component of the Pathways transfer guarantees is working smoothly. The analysis examined data from 1,970 CUNY students who have completed at least one Pathways Major Gateway course, transferred to another CUNY college in Spring 2016, and declared one of the participating majors. The transfer credit for these Major Gateway courses should have been applied as credit toward the major or as fulfilling prerequisite requirements for the major. To investigate whether the transfer guarantee was applied correctly, an analysis of the actual transferred equivalencies at the receiving institution was conducted. Table 4 shows that 86.9 percent of all incoming Major Gateway courses were accepted as equivalent Major Gateway courses and were counted toward the major at the 10

receiving colleges. Moreover, 10.6 percent of all incoming Major Gateway courses were accepted as other credit bearing courses applicable to students majors. Overall, 97.5 percent of all incoming Major Gateway courses were accepted as credit applicable to students majors, suggesting a successful implementation of the Major Gateway transfer guarantees. Column definitions for Tables 4 and 5 are as follows: Major Gateway is the course transferring as equivalent to a Major Gateway course at the receiving college; Other Course is the course transferring as equivalent to a course not identified as a Major Gateway at the receiving college, for example, ENG 111 (Introduction to College Writing); Subject Elective is the course transferring for elective credit within a particular discipline, for example, BIOL 1000E (Biology Elective); Blanket Elective is the course transferring for blanket elective credit, for example, ELEC 1000 (Elective Credit). Table 4 Transfer Status of Incoming Major Gateway Courses at Receiving Colleges, Spring 2016 Receiving Colleges Total Number of Major Gateway Courses 1 Transfer Status of Incoming Major Gateway Courses at Receiving Colleges Major Gateway Other Course Subject Elective Blanket Elective N % % % % Senior 4,427 87.5 10.1 1.9 0.6 Comprehensive 327 76.1 20.5 2.1 1.2 Community 2 223 90.6 7.6 1.3 0.4 University Total 4,977 86.9 10.6 1.9 0.6 Data Source: Institution Research Database (IRDB) and CUNYfirst 1 Major Gateway courses taken at a sending institution and transferred into a receiving institution in Spring 2016. 2 Because two community colleges conduct credit evaluations manually, their data are not included in this table. Only automatic credit evaluations are presented in current analysis. Table 5 included the analysis for the ten participating majors. In Criminal Justice, Sociology, and Teacher Education, Major Gateway courses taken at sending colleges are less likely to transfer as equivalent to Major Gateway courses at receiving college than in other disciplines. This is due to the more flexible categories chosen by these faculty committees. It is important to note, however, that the transfer credits evaluated as Subject Elective or Blanket Elective are also relatively low for Criminal Justice and Sociology. This analysis points to areas where further research and investigation is necessary to improve transfer within the majors. 11

Table 5 Transfer Status of Incoming Major Gateway Courses at Receiving Colleges, by Major Majors Total Number of Major Gateway Courses 1 Transfer Status of Incoming Major Gateway Courses at Receiving Colleges Major Gateway Course Subject Elective Blanket Elective N % % % % Biology 596 89.9 7.0 2.9 0.2 Business 2,003 93.5 5.1 1.3 0.1 Criminal Justice 129 65.9 32.6 0.8 0.8 Economics 97 76.3 16.5 7.2 0.0 English 200 74.5 20.0 3.5 2.0 Nursing 540 87.6 11.3 0.6 0.6 Political Science 65 80.0 15.4 1.5 3.1 Psychology 923 88.0 10.8 1.0 0.2 Sociology 104 63.5 34.6 1.9 0.0 Teacher Education 320 63.8 24.7 6.9 4.7 Total 4,977 86.9 10.6 1.9 0.6 Data Source: Institution Research Database (IRDB) and CUNYfirst 1 Major Gateway courses taken at a sending institution and transferred into a receiving institution in Spring 2016. Student Appeals Process Review The Pathways resolution called for the establishment of an appeals process for undergraduate students who wish to appeal denial or restriction of transfer credit. In response, each college appointed an appeals officer and established a student appeals process. The Central Office of Academic Affairs (OAA) also adopted procedures for handling student appeals that had received a negative determination at the campus level. Guidelines and procedures were developed and posted on the CUNY Pathways website 5 and on each college website. In June and July 2016, CUNY Central OAA surveyed 18 CUNY colleges regarding their student appeals processes and outcomes since the implementation of Pathways (i.e., academic years 2013-14, 2014-15, and 2015-16). More specifically, colleges were asked about their Pathways Appeals websites for students, their appeals processes and tracking systems, the numbers and types of appeals they received, and problems and issues they encountered. Appeals Survey Findings Transparency and visibility of appeals processes. Fourteen of the 18 colleges provided links (URLs) to websites describing some information about their local appeals processes. Of the 14, 11 colleges websites provide detailed information about processes; three others show either contact information of 5 http://www2.cuny.edu/about/administration/offices/undergraduate-studies/pathways/rightsandresponsibilities/ 12

the transfer appeals officer or advisement office, or a link to the appeals form. Two of the remaining four colleges provided links to websites that direct students to Central OAA s website with appeals information. One college s link directs students to their Pathways webpage with no information about the appeals process. The other one had a broken link. All 18 colleges, except the last one, provide students with information on the University appeals process. Appeals processes and tracking systems. There is broad variation in how the appeals process is implemented at each college. Currently, 10 colleges have established committees to review appeals, with only one including a student representative, while at six others the appeals officer manages the process, escalating it as needed to an administrator, department chair, or advisor. Time to completion of the appeals process takes, on average, 15 business days but could be longer depending on the nature of the appeal and when it is received. Two colleges do not have an established appeals process. In terms of tracking systems, half of the 18 respondents said they have established systems. Some of them described the information that is collected, but few included information about who/which office collects the information or where the information is stored. The remaining nine colleges indicated either that they had no tracking system or provided no information. The numbers, types, and patterns of appeals. Of the 18 colleges, seven received no appeals and three received fewer than 100 appeals over the last three academic years. In contrast, three colleges received a large number of appeals: each had more than 1,000 over the three-year period, including one college with more than 2,500. Of the remaining five colleges, three indicated that the number of appeals received was unknown, and two did not provide this information. Such variability in the number of appeals across colleges may be because there has been no uniform definition of what constitutes a Pathways appeals. For example, at one college, most of the more than 1,000 appeals were cases where returning students opted out of Pathways to fulfill their degree requirements and graduate sooner. It is unknown whether other colleges experienced such situations, and if so, whether or not they were counted as Pathways appeals. In addition, since most colleges did not track appeals in a consistent manner, it was not possible to identify underlying patterns in appeals across the University. Recommendations Regarding Student Appeals Processes The survey suggests that Pathways appeals practices vary a great deal across the University. A few colleges practices stood out as noteworthy. For example, two colleges reported that their appeals review committees have regularly scheduled meetings. One has a clearly outlined appeals process and the committee, chaired by the appeals officer, includes broad representation from faculty and administrative offices such as the registrar, admissions, and advisement. One college reported that they have tried to reduce the number of appeals by improving instruction, and another college conducted an analysis of the general education courses at their largest non-cuny feeder institution. Other recommendations emerged from the survey, such as the need for a University-wide definition of Pathways appeals and a common appeals tracking system. This should be achieved through a collaboration between the colleges and the Central Office. Second, each college should establish a clear Pathways appeals process, including the convening of an appeals review committee and the 13

dissemination of information so that students will understand the process better. Third, the Central Office should clarify the CUNY-level appeals process, compile an annual report based on the collective data across the university on the appeals, and share it with the colleges. 14

Early Outcomes of the Pathways Initiative Efficiency of Credit Transfer For students who transfer within CUNY, the Pathways transfer guarantees more credits transferred to receiving colleges. Furthermore, the Pathways Initiative reduced the number of general education requirements at many CUNY colleges to 42 credits, a reduction from approximately 60 credits at several colleges. Table 6 shows a steady increase, on average, of credits transferring to receiving colleges after the Fall 2013 Pathways implementation; the mean number of credits transferred at senior colleges was 56.4 in Fall 2012, while the number increased to 61.5 in Fall 2015. For community colleges, the numbers were 15.7 in Fall 2012 and 20.4 in Fall 2015. The table also shows that, after one year, students consistently earned approximately 20 additional credits, which indicates that both before and after Pathways, students momentum during their first year after transfer was stable. Table 6 Credits Transferred for and Accumulated by Transfers Within CUNY 1 Fall 2012 Fall 2013 Fall 2014 Fall 2015 Receiving Colleges Enrolled Credits Transferred Credits after One Year Enrolled Credits Transferred Credits after One Year Enrolled Credits Transferred Credits after One Year Enrolled Credits Transferred Credits after One Year N Mean Mean N Mean Mean N Mean Mean N Mean Mean Baruch 929 58.7 79.8 1,012 65.2 86.1 1,277 61.5 82.1 1,172 62.9 83.4 Brooklyn 1,086 59.8 81.4 1,266 58.0 78.8 1,460 58.6 79.9 1,587 58.1 79.2 City 795 57.2 76.9 795 52.4 72.0 937 61.0 80.9 970 63.2 83.4 Hunter 993 56.6 77.4 1,043 61.0 81.1 1,183 62.0 82.4 1,099 61.3 82.1 John Jay 870 59.4 80.5 955 59.8 81.6 926 62.2 83.9 1,092 63.5 84.7 Lehman 780 63.2 82.8 1,021 63.7 83.3 1,132 62.3 81.6 1,264 61.3 80.5 Medgar Evers 245 49.0 66.3 252 53.0 71.0 268 50.6 67.8 284 58.8 77.0 NYCCT 663 39.7 58.4 671 49.1 66.8 802 58.5 76.5 889 60.8 78.9 Queens 1,179 62.4 83.1 1,274 60.8 81.2 1,528 63.9 84.4 1,502 64.6 85.2 Staten Island 822 41.9 62.3 202 43.9 62.0 258 47.6 67.1 332 54.1 76.7 York 513 59.9 79.0 472 59.5 79.1 502 59.5 78.4 579 59.3 78.9 Professional Studies 120 62.6 77.1 130 59.6 74.8 162 63.8 78.4 222 66.0 80.0 Senior College 2 Total/Average 8,995 56.4 76.8 9,093 59.0 79.1 10,435 60.8 80.9 10,992 61.5 81.7 BMCC 174 13.4 26.6 264 21.6 37.4 492 20.1 34.1 443 28.1 43.0 Bronx 232 12.2 26.2 331 13.6 26.4 323 15.1 29.5 335 16.6 31.3 Guttman -- -- -- -- -- -- -- -- -- -- -- -- Hostos 74 10.1 27.4 184 9.0 23.3 178 15.4 29.4 326 14.7 30.5 Kingsborough 639 18.1 35.9 786 17.9 35.9 690 18.4 35.4 811 20.1 38.0 LaGuardia 449 17.1 31.1 511 16.0 30.1 590 17.8 32.5 585 18.7 35.3 Queensborough 215 13.7 28.5 255 16.6 30.6 359 21.5 36.3 434 22.5 37.7 Community College Total/Average 1,783 15.7 31.3 2,331 16.4 31.8 2,632 18.4 33.5 2,934 20.4 36.6 Total 10,778 50.5 70.1 11,424 51.3 70.5 13,067 53.2 72.4 13,926 53.7 73.1 Data Source: Institutional Research Database (IRDB) 1 Consists of students enrolling as a transfer student in the fall semester less than three years since their last enrollment at CUNY. 2 Includes the three comprehensive colleges 15

GPA and Credit Attainment after One Year Table 7 includes GPA and the number of credits earned as of the end of first year by first-time freshmen, along with those by transfer students. All four cohorts, before and after the Fall 2013 implementation of Pathways, show stable GPAs in each group. However, while the number of credits after one year for first-time freshmen were relatively unchanged, students who transferred within CUNY had a steady increase in accumulated credits. One side note is that transfer students consistently have a higher GPA one year later than first-time freshmen after their first year, both at senior and community colleges. These outcomes indicate transfer students perform better in more advanced courses during the first year after transfer, than first-time freshmen in entry-level courses during their first year. Table 7 GPA and Credit Accumulation of First-Time Freshmen and Transfers Within CUNY Groups Fall 2012 Fall 2013 Fall 2014 Fall 2015 First-Time Freshmen at Senior Colleges 2 Enrolled (N) 17,182 17,880 18,053 18,413 GPA after One Year 2.74 2.74 2.74 2.77 Credits after One Year 25.3 25.1 25.5 26.2 Transfer to Senior Colleges 2 Within CUNY 1 Enrolled (N) 8,995 9,093 10,435 10,992 GPA after One Year 2.83 2.83 2.81 2.84 Credits after One Year 76.8 79.1 80.9 81.7 First-Time Freshmen at Community Colleges Enrolled (N) 18,434 17,742 19,322 19,022 GPA after One Year 2.29 2.33 2.28 2.34 Credits after One Year 15.1 15.9 15.7 15.9 Transfer to Community Colleges 2 Within CUNY 1 Enrolled (N) 1,783 2,331 2,632 2,934 GPA after One Year 2.44 2.50 2.47 2.59 Credits after One Year 31.3 31.8 33.5 36.6 First-Time Freshmen - Total Enrolled (N) 35,616 35,622 37,375 37,435 GPA after One Year 2.52 2.54 2.51 2.56 Credits after One Year 20.2 20.7 20.7 21.2 Transfer Within CUNY - Total Enrolled (N) 10,778 11,424 13,067 13,926 GPA after One Year 2.78 2.77 2.75 2.79 Credits after One Year 70.1 70.5 72.4 73.1 Data Source: Institutional Research Database (IRDB) 1 Consists of students enrolling as a transfer student in the fall semester less than three years since their last enrollment at CUNY. 2 Includes the three comprehensive colleges 16

Retention Rates and Transfer Type One-year retention rates have remained consistent since before Pathways was implemented, as shown in Table 8. These retention rates include students enrolled at any CUNY college in the following Fall semester, including students who transferred within the CUNY system, before earning the degree pursued. Table 8 One-Year Retention Rates 1 of First-Time Freshmen Fall 2011 Fall 2012 Fall 2013 Fall 2014 Degree Pursued Enrolled Retained Enrolled Retained Enrolled Retained Enrolled Retained N % N % N % N % Associate Total 24,865 64.8 23,630 65.2 23,452 66.4 24,681 64.4 Baccalaureate Total 12,000 86.2 11,836 86.3 12,007 87.1 12,521 86.4 Data Source: Institutional Research Database (IRDB) 1 Retention rates are the percentage of students who are still enrolled at any CUNY college in the subsequent fall term and have not yet earned the degree pursued. The number of transfers at CUNY is on the rise. This increase, as Table 9 shows, is mostly due to an increase in the number of students with associate degrees who transfer to a baccalaureate program, especially after Pathways implementation in Fall 2013. Table 9 Trends in Transfers into CUNY Baccalaureate Programs Transfer Type Fall 2012 Fall 2013 Fall 2014 Fall 2015 % % % N N From Within CUNY increase 1 N increase 1 N increase 1 With Associate Degree 4,030 4,158 3.2 4,803 19.2 5,293 31.3 Without Associate Degree 2,456 2,078-15.4 2,571 4.7 2,696 9.8 From Baccalaureate Program 1,545 1,586 2.7 1,866 20.8 1,877 21.5 From CUNY More than 3 Yrs 2 891 908 1.9 1,031 15.7 1,172 31.5 From Outside CUNY 5,548 5,628 1.4 5,650 1.8 5,759 3.8 Total 14,470 14,358-0.8 15,921 10 16,797 16.1 Data Source: Institutional Research Database (IRDB) 1 % increase from Fall 2012 2 Students enrolling as a transfer student in the fall semester more than three years since their last enrollment at CUNY. Course-Taking Patterns by Discipline Each Pathways Common Core area accommodates courses from a wide array of disciplines. As a result, specific courses that had been required in college-specific general education curricula prior to Pathways could still be offered as general education compliant courses within Pathways new flexible structure. With this in mind, CUNY s Office of Institutional Research and Assessment (OIRA) examined the number and percentage of first-time freshmen taking courses in particular disciplines during their first year at CUNY. The analysis includes a comparison of first-time freshmen enrolled at CUNY before and after the 17

Pathways implementation in Fall 2013. Table 10 shows these course-taking patterns vary widely across disciplines. The percentages of students who took a course in some disciplines have remained fairly consistent. For example, course-taking of Foreign Language only ranged between 18.1 percent (Fall 2012 entrants) and 20.7 percent (Fall 2014) across the five-year period. Some other disciplines, including Business, History, and Social Sciences, also show fairly consistent course-taking patterns. In contrast, course-taking in disciplines including Math and Natural Sciences has steadily increased. The percentage of students who took a course in Math increased from 77.9 percent for the Fall 2011 cohort to 86.3 percent for the Fall 2015 cohort. Respective numbers for Natural Sciences are 27.0 percent and 35.1 percent. Course-taking in disciplines such as Arts, Education, and Speech/Communication, on the other hand, has steadily declined throughout the five years; for instance, 44.9 percent of the Fall 2011 entrants took a course in Arts, but the number decreased to 34.7 percent for the Fall 2015 entrants. A close look at the table reveals that both of these changes may have started before the Fall 2013 Pathways implementation. It is possible that multiple variables are responsible for them, which may include federal and state policy changes, and national trends in terms of increasingly popular/unpopular disciplines and occupations. Table 10 Trends in First-Time Freshman Course-Taking, by Discipline Discipline Data Source: Institutional Research Database (IRDB) 1 Proportion of first-time freshmen who took a course in a given discipline in the first fall or spring semester after entrance 2 Spring 2016 data are preliminary Fall 2011/ Spring 2012 Took Course during the Academic Year 1 Fall 2012/ Fall 2013/ Fall 2014/ Fall 2015/ Spring 2013 Spring 2014 Spring 2015 Spring 2016 2 N % N % N % N % N % Architecture 233 0.6 188 0.5 203 0.6 176 0.5 199 0.5 Arts 16,678 44.9 15,802 44.4 15,186 42.6 14,790 39.6 12,997 34.7 Business 5,892 15.9 5,685 16.0 5,466 15.3 5,860 15.7 5,540 14.8 Computer Science 3,318 8.9 3,337 9.4 3,254 9.1 3,574 9.6 3,548 9.5 Criminal Justice 2,049 5.5 1,949 5.5 2,307 6.5 2,592 6.9 2,381 6.4 Education 4,418 11.9 3,791 10.6 2,594 7.3 2,463 6.6 2,246 6.0 Engineering/Technology 1,568 4.2 1,595 4.5 1,648 4.6 1,731 4.6 1,718 4.6 Foreign Languages 7,384 19.9 6,454 18.1 6,965 19.6 7,736 20.7 7,260 19.4 History 8,906 24.0 8,461 23.8 8,816 24.7 8,999 24.1 8,988 24.0 Humanities 34,394 92.6 33,546 94.2 33,562 94.2 35,209 94.2 33,392 89.2 Math 28,930 77.9 28,363 79.6 29,745 83.5 31,518 84.3 32,323 86.3 Natural Sciences 10,021 27.0 9,863 27.7 12,108 34.0 12,723 34.0 13,145 35.1 Nursing/Health 6,659 17.9 5,984 16.8 4,342 12.2 4,033 10.8 3,581 9.6 Social Sciences 28,468 76.7 27,794 78.0 27,793 78.0 28,948 77.5 29,027 77.5 Speech/Communication 14,085 37.9 13,460 37.8 11,179 31.4 11,509 30.8 11,734 31.3 Total First-time Freshmen 37,129 35,616 35,622 37,375 37,446 18

OIRA also examined course-taking patterns of all undergraduates enrolled in the Fall semester who took courses in particular disciplines during the Fall and Spring semesters of the academic year (Table 11). This analysis includes students who were following the Pathways curriculum, as well as students who were following the prior curricular requirements. Course-taking among all undergraduates across disciplines has varied only slightly, with the percentages mostly remaining level throughout the five years. Although a few disciplines, namely, Arts, Education, and Nursing/Health, show mildly declining patterns, these changes started before the Fall 2013 implementation of Pathways; they may also be attributable to some variables including or other than the Pathways Initiative. Table 11 Trends in Undergraduate Course-Taking, by Discipline Took Course during the Academic Year* Discipline Data Source: Institutional Research Database (IRDB) 1 Proportion of undergraduates enrolled in the fall semester who took a course in a given discipline in a fall or spring semester 2 Spring 2016 data are preliminary Fall 2011/ Spring 2012 Fall 2012/ Spring 2013 Fall 2013/ Spring 2014 Fall 2014/ Spring 2015 N % N % N % N % N % Architecture 1,307 0.5 1,189 0.5 1,172 0.5 1,169 0.5 1,109 0.5 Arts 66,742 27.9 62,980 26.5 63,737 26.6 63,057 25.7 56,921 23.3 Business 47,141 19.7 46,769 19.7 47,593 19.9 48,446 19.7 47,063 19.2 Computer Science 23,711 9.9 24,262 10.2 24,372 10.2 25,921 10.6 25,979 10.6 Criminal Justice 11,516 4.8 11,937 5.0 13,197 5.5 13,683 5.6 13,382 5.5 Education 26,190 11.0 24,133 10.2 21,131 8.8 18,909 7.7 17,947 7.3 Engineering/Technology 9,141 3.8 8,981 3.8 9,535 4.0 10,135 4.1 10,411 4.3 Foreign Languages 48,184 20.2 46,313 19.5 46,272 19.3 48,451 19.7 47,430 19.4 History 42,990 18.0 42,742 18.0 43,932 18.3 43,769 17.8 41,549 17.0 Humanities 132,203 55.3 130,793 55.0 129,738 54.2 130,792 53.2 122,856 50.2 Math 105,814 44.3 105,235 44.3 107,868 45.0 110,615 45.0 108,283 44.2 Natural Sciences 77,208 32.3 79,449 33.4 82,810 34.6 88,568 36.1 87,356 35.7 Nursing/Health 36,733 15.4 35,632 15.0 33,297 13.9 31,024 12.6 28,814 11.8 Social Sciences 150,587 63.0 150,000 63.1 150,808 63.0 155,085 63.1 153,299 62.6 Speech/Communication 39,955 16.7 39,100 16.4 36,345 15.2 37,732 15.4 36,692 15.0 Total Undergraduates 239,103 237,737 239,497 245,646 244,782 Fall 2015/ Spring 2016** 19

Next Steps Further Review and Analysis The University will undertake further review and analysis to address the following issues: Proper and consistent implementation. The University continues to undertake a systematic review of Pathways implementation to ensure the policies are correctly and consistently translated into practices for the benefit of students. Student communications. This evaluation, along with previous and ongoing reviews of the Pathways Initiative, indicates that further work is needed to ensure students understand their curriculum requirements and the University s transfer guarantees. The Central Office and the colleges could do more to better inform students about their curriculum, general education in general, and the avenues available to them to appeal decisions that are in violation of the Pathways transfer guarantees. Course Review and Submission process. To promote a greater understanding among faculty and to provide additional guidance and direction, the process for submitting Pathways courses for review to the faculty Common Core Course Review Committee (CCCRC) could be more transparent. Student course-taking patterns. Prior to the Pathways implementation, there was a concern regarding possible changes in student course-taking due to Pathways. However, to date, no remarkable changes have been observed; while there have been small changes, it is unknown whether they are attributable to Pathways or a reflection of national trends. The University will continue to collect and analyze student course-taking to determine whether there are any noteworthy trends associated with Pathways. Student appeals process. There is a need for CUNY to create a University-wide definition of Pathways appeals and to develop a consistent appeals tracking system. This should be achieved through a close collaboration and communication between the colleges and the Central Office. Each college should establish a clear Pathways appeals process, including the convening of an appeals review committee and the dissemination of information so that students will understand the process better. College Option. Questions have been raised about fulfillment of College Option requirements, in particular, related to the frequency that courses taken at the sending college transfer as meeting College Option requirements at the receiving college. These issues will be examined and results reported in future reports. Gateway Majors. This analysis points to areas where further research and investigation is necessary to improve transfer within majors. There are currently ten popular majors that have 20

identified courses guaranteed to transfer for credit toward the major. The next step will be to increase the number of participating majors. This report was compiled by the Central Office of Academic Affairs with contributions from the Year-Two Faculty Review Team, the chairs and faculty members of the Common Core Course Review Committee, the college Pathways appeals officers, and the student focus group participants. The 2011 Pathways Resolution requires that the Pathways policies and processes be reviewed and evaluated each year for three years beginning in 2013, and every three years thereafter, to modify them as necessary to improve them or to meet changing needs. The Central Office of Academic Affairs will continue to monitor, track, and make public Pathways data on an annual basis. While the next mandated review is scheduled for 2019-2020, efforts are underway to explore options for a comprehensive external review and analysis of Pathways implementation and effectiveness. 21