ALABAMA ARTICULATION AND GENERAL STUDIES COMMITTEE AND STATEWIDE TRANSFER/ARTICULATION REPORTING SYSTEM EVALUATION PROJECT FINAL REPORT W. Donald Crump Project Coordinator Dr. Marcia O Neal Dr. Peggy Wilds Project Evaluators Not everything that counts can be counted, and not everything that can be counted counts. Albert Einstein Submitted To: Articulation and General Studies Committee Montgomery, Alabama May 22, 2002
FOREWORD Evaluation, by definition, is an effort to determine the worth or value of an activity. While numbers can be useful in an evaluation, additional sources of information are essential in providing a sound basis for judgments that are integral to an effective evaluation process. Many people in different roles and responsibilities have contributed their time, energy, thoughts, opinions, and judgments to this evaluation project. Any contributions this project may make to improve the general studies curriculum and the articulation and transfer process in Alabama are due primarily to those individuals who were willing to take time and openly share their viewpoints and suggestions. A special appreciation goes to those students, advisors, faculty, and administrators who participated in the surveys and in focus group discussions. Their enthusiasm, openness, and candor made these phases of the project a much more pleasant and productive experience. Appreciation is expressed also to those presidents, former AGSC members, and legislators who participated in interviews and provided much valuable insight. Many individuals provided valuable assistance, support, and advice during the project. Members of the AGSC offered helpful guidance and direction in the planning phase and provided useful feedback during the project. Dr. Elizabeth French from the Alabama Commission on Higher Education provided a useful summary of the legislation and accomplishments of the AGSC. The minutes of the AGSC maintained so carefully by Ms. Deborah Nettles, also from the Alabama Commission on Higher Education, provided a useful reference for several aspects of the project. Community college and university representatives were helpful in arranging for focus group meetings on their campuses. Mr. Shannon Nichols and Ms. Anita Walker and other members of the STARS staff always were prompt and efficient in responding to requests for information and assistance. A special thanks goes to Ms. Carmen Johnson for her valuable assistance in putting together the final report and for her patient and persistent attention to details. Dr. Marcia O Neal and Dr. Peggy Wilds were superb in their contributions to the evaluation project. They each brought invaluable knowledge and experience to the project and their high degree of professional commitment was evident in the manner in which they fulfilled their responsibilities throughout the project. Their friendship, advice and encouragement added much to the value of the final report. W.D.C. ` May 2002 i
TABLE OF CONTENTS Foreword... i Executive Summary... v Introduction... 1 Articulation and General Studies Act... 2 Accomplishments of the AGSC and STARS... 3 Rationale for the AGSC/STARS Evaluation Project... 7 AGSC/STARS Evaluation Project Goals, Design, and Methods... 7 Questionnaire Surveys... 8 Focus Groups... 9 In-Person and Telephone Interviews... 9 Results for the AGSC/STARS Evaluation Project... 11 Results from Surveys of Students... 11 Results from Surveys of Advisors et al... 15 Summary and Comparison of Student and Advisor Survey Results... 21 Results from Survey of Transfer Contact People... 24 Results from Focus Group Discussions... 28 Results from In-Person and Telephone Interviews... 32 Discussion, Conclusions, and Implications... 34 How Are We Doing... 34 Where Do We Go from Here... 35 Achievement of Legislative Intent... 36 Evaluation of AGSC Organizational Structure... 36 Effectiveness of AGSC... 37 Evaluation of Impact of Act 94-202 on Higher Education... 37 Recommendations... 38 List of Appendixes... iii List of Tables... iv List of Figures... iv ii
LIST OF APPENDIXES Appendix A Accomplishments of AGSC/STARS... A-1 Appendix B Survey Forms... B-1 Appendix C Information about Student Survey Respondents... C-1 Appendix D Information about Advisors, Faculty, Administrators Respondents... D-1 Appendix E Information about Transfer Contact Group Respondents... E-1 Appendix F Listing of Focus Group Discussions... F-1 Appendix G Compilation of Comments from Student Survey... G-1 Appendix H Descriptive Statistics for Students on Selected Comparisons... H-1 Appendix I Compilation of Comments from Advisors et al... I-1 Appendix J Descriptive Statistics for Advisors et al on Selected Comparisons... J-1 Appendix K Compilation of Comments from Community College Students Focus Group Discussions... K-1 Appendix L Compilation of Comments from University Transfer Students Focus Group Discussions... L-1 Appendix M Compilation of Comments from Community College Advisors et al Focus Group Discussions... M-1 Appendix N Compilation of Comments from University Transfer Officials Focus Group Discussions... N-1 Appendix O Compilation of Comments from Interviews... O-1 iii
LIST OF TABLES Table 1. STARS Usage by Academic Year... 4 Table 2. STARS Usage by Two-Year Institutions from July December 2001... 6 Table 3. Objectives for Focus Group Discussions... 10 Table 4. Student Survey Results for Total Group... 11 Table 5. Survey Results for Advisors, Faculty, Administrators, and Others... 15 Table 6. Comparison of Evaluation Results for Students and Advisors... 21 Table 7. Factors Affecting Increase in Transfer Students... 24 Table 8. Factors Contributing to Transfer Appeals... 25 Table 9. Receipt and Use of Monthly STARS Reports at Universities... 26 Table 10. Impact of AGSC/STARS on Student Transfer Process... 27 Table 11. Key Insights and Suggestions from Community College Focus Group Discussions... 28 Table 12. Key Insights and Suggestions from University Transfer Students Focus Group Discussions... 29 Table 13. Key Insights and Suggestions from Community College Advisors et al Focus Group Discussions... 30 Table 14. Key Insights and Suggestions from University Transfer Appeals Officials Focus Group Discussions... 31 LIST OF FIGURES Figure 1. STARS Usage from November 2000-February 2002... 5 iv
EXECUTIVE SUMMARY The articulation of degree program requirements and the opportunity for transfer of academic credit are crucial considerations for many students in achieving their goals in higher education in Alabama. Act 94-202, the articulation and general studies act, created the Articulation and General Studies Committee (AGSC) to implement specific provisions of the legislation and designated a Statewide Transfer and Articulation Reporting System (STARS). The AGSC developed a general studies curriculum and an articulation process within the first few years of activity and STARS became a statewide information system in the summer of 1998. June 2001 marked the completion of the seventh year of the AGSC and the third year of STARS as a statewide information system for facilitating the articulation of degree program requirements and the transfer of academic credit. At that time, the AGSC determined the need for an evaluation to address key questions and concerns related to the legislation, the organization and responsibilities of the AGSC, and the impact of articulation and the general studies curriculum on higher education in Alabama. The AGSC/STARS Evaluation Project was implemented to address two key questions: How are we doing? Where do we go from here? The AGSC identified four goals to give direction to the design and methods to be employed in the evaluation project. These goals were: 1. To determine the degree to which the intent of Act 94-202 has been achieved; 2. To evaluate the effectiveness of the organizational structure established by the statute; 3. To assess the degree to which AGSC is fulfilling its responsibilities; and 4. To evaluate the impact of the articulation process on higher education in Alabama. The charge given by the AGSC emphasized the importance of addressing the major questions and goals of the evaluation project from multiple perspectives and stressed including those groups using STARS as a part of the evaluation process. The evaluation design incorporated both quantitative and qualitative methods. The primary evaluation strategies included: 1. Questionnaire surveys to students; to community college and university advisors, faculty, administrators, and others; and to college and university transfer contact people; 2. Focus group discussions with community college students, university transfer students, community college advisors, and university transfer appeals officials; 3. Interviews with a sample of college/university presidents or chief academic officers, former AGSC members, and legislators; and 4. Data from the STARS system. Extensive data and information from surveys, focus groups, and interviews are provided in the report. Evaluation results from practically all sources provide strong support for the AGSC and STARS. Survey results from all three groups overwhelmingly supported STARS both in terms of the ratings and the comments that were offered. Focus groups of v
both community college and university transfer students were enthusiastic in their comments about STARS. Community college advisors, faculty and administrators participating in focus groups were strong advocates for STARS, the general studies curriculum, and the articulation of degree programs. The focus group of university transfer appeals officials also supported the value of STARS in facilitating the transfer process and reducing significantly the number of transfer appeals. Participants acknowledged several additional benefits from STARS and the statewide articulation of degree programs. Results from interviews with presidents, former AGSC members, and legislators provided additional support for the accomplishments of the AGSC. Overall, representatives from all groups cited the legislation and the achievements of the AGSC and STARS as instrumental in creating a climate of cooperation and communication between community colleges and state universities. Some sentiment among a small number continues to exist, however, that the impact of Act 94-202 has been to diminish the quality of higher education in Alabama. The data and comments from the surveys, the comments and suggestions for improvement from the focus groups, and the comments from interviews identified a number of concerns to be addressed by the AGSC and STARS. Area V related to pre-professional/pre-major studies was the foremost topic of concern for all groups. Additional topics of concern are included in the report as well as important implications for the organization of the AGSC and STARS. Based on the results and discussion, the following twelve interrelated recommendations are offered: 1. Improvement of Area V pages should be a top priority of the AGSC and STARS. 2. The AGSC needs to establish an executive position to fulfill administrative responsibilities and serve as the designated contact person for the committee and STARS. 3. Improved coordination and communication about curricular and degree program changes are needed among academic committees, universities, and STARS. 4. The STARS website should be revised with primary emphasis given to use by students and advisors. 5. Monthly reports to universities need refinement. 6. Efforts to increase the visibility of STARS statewide are needed. 7. STARS should increase outreach services to ensure greater participation by all community colleges and state universities. 8. Community colleges should systematically introduce STARS to students early in their first semester of enrollment. 9. Careful review of the alignment of associate degree requirements with those for the baccalaureate degree should be completed. vi
10. The AGSC should not pursue a statewide uniform course numbering and title system. 11. The relationship between the AGSC and STARS needs clarification. 12. The AGSC needs to conduct a carefully designed study to evaluate the success of community college students to state universities. vii
ALABAMA ARTICULATION AND GENERAL STUDIES COMMITTEE AND STATEWIDE TRANSFER AND ARTICULATION REPORTING SYSTEM EVALUATION PROJECT FINAL REPORT Not everything that counts can be counted, and not everything that can be counted counts. Albert Einstein Introduction Higher education is a major commitment of state government and the citizens of Alabama. The state s higher education institutions include 31 community and technical colleges and 17 public universities. The community and technical college system is administered through the Alabama Department of Postsecondary Education and its governing board is the State of Alabama Board of Education. Among the 17 public universities, nine are administered through three university systems each with a governing board and the remaining eight each have separate governing boards. The combined budgets for these institutions exceed $2,650,000,000. Over 195,000 students are enrolled in Alabama s public higher education institutions with approximately 72,000 in community and technical colleges and 123,000 in state universities. Among this number, approximately 11,500 students transferred between and among community colleges and universities in the 2000-2001 academic year. Approximately 5,000 of this number transferred from community colleges to state universities and, surprisingly, almost 2,800 students transferred from state universities to community colleges. Given that the average length of college enrollment is five years or more and the number of transfer students is 11,500 per year, this suggests that over 55,000 of the students enrolled in Alabama s public higher education institutions have transferred from one institution to 1
another in their academic careers. The articulation of degree program requirements and the opportunity for transfer of academic credit are crucial considerations for many students in achieving their goals in higher education in Alabama. The establishment and growth of community and technical colleges in Alabama have occurred over the past three decades. In the beginning years, transfer from community colleges to universities was based on well-established degree program requirements and articulation arrangements between individual community colleges and state universities as the need might arise. In the 1980s, universities began establishing core curricular requirements in general studies for the freshman and sophomore years. Professional programs (e.g., engineering, business, education, nursing) began pushing specific requirements for the major field into the first two years of collegiate study. The diversity of general studies requirements at public universities in Alabama required many community colleges to offer a wide range of courses in the humanities, mathematics, social sciences, and natural sciences in order to enable students to transfer to the university of their choice. The combination of these factors caused significant problems for the articulation of degree program requirements between community colleges and public universities and many students encountered difficulty in transferring academic credit in pursuit of their goals in higher education. Articulation and General Studies Act The Alabama Legislature passed Act 94-202, frequently referred to as the articulation and general studies act, in March 1994. The primary goals of the act were: To provide for a uniform articulation agreement among all institutions of higher education as well as a statewide general studies curriculum; To provide for the computation of grade point averages for certain transferred students; To specify that the Act would not impede the objectives of historically black institutions; To provide for implementation conditioned on the participation of certain institutions; and To specify certain reporting requirements. The Act provided for the creation of an Articulation and General Studies Committee (AGSC) to implement specific provisions included in the legislation. Membership on the committee, the appointment process, and the rotation of leadership were defined in the legislation. The Act also designated the agency responsible for developing the computerized database and designing student contractual agreements to be honored among all public institutions. Specific tasks assigned to the AGSC were: To develop a statewide general studies curriculum no later than September 1, 1998; To develop and adopt an articulation agreement by September 1, 1999 for the transfer of credit among all public institutions of higher education in Alabama; To examine the need for a uniform system for course numbering, titles, and descriptions; and To resolve problems in the administration or interpretation of the articulation agreement of the general studies curriculum. 2
The 10-member AGSC began its work in July 1994 and has included alternates to serve in the absence of appointed members. The Executive Director of the Alabama Commission on Higher Education and the Director of Articulation for the Statewide Articulation and Transfer Reporting System serve as non-voting members of the AGSC. Academic Committees were established by the AGSC to support the development of the general studies curriculum and continue to be used in dealing with on-going curriculum issues. The Statewide Transfer and Articulation Reporting System (STARS) designated in the legislation is located at Troy State University. STARS is a web-based data system that allows students, advisors, faculty, and administrators of all community and technical colleges and universities as well as the general public access to information about the general studies curriculum, articulation of degree program requirements, and the transfer of academic credit. Students using this system can obtain a transfer guide that includes an agreement that obligates the receiving state university to honor the equivalent credits identified. The STARS office is involved in the day-to-day operations of the AGSC and in coordinating the activities of the Academic Committees. STARS serves as a collection agency and a clearinghouse for all information concerning the statewide general studies curriculum. Guidelines and course requirements determined by the AGSC are maintained through personnel and resources of the STARS office. Accomplishments of the AGSC and STARS The AGSC and STARS have made significant accomplishments in response to each of the four tasks identified in the previous section since the initial formation of the committee in July 1994. Foremost among these accomplishments was the adoption of the framework for the statewide general studies curriculum on December 6, 1995, within the first 18 months of the committee s formation. This framework consisted of a general studies component which includes Areas I through IV: written composition; humanities and fine arts; history, social, and behavioral sciences; and natural sciences and mathematics. The framework also includes Area V, which is a pre-professional/pre-major component. Subsequently, consensus was achieved on the process for the development of the preprofessional/pre-major component for the articulation of degree program requirements. An AGSC Technology Task Force worked with the STARS staff in developing institutional homepages for each of the state universities and the elements of the transfer process were established. An articulation agreement and process for implementation were developed and adopted by the AGSC by September 1, 1999. A comprehensive summary of the accomplishments of the AGSC and STARS through September 2000 was prepared by Dr. Elizabeth French of the Alabama Commission on Higher Education. Additional detail and information about these accomplishments are provided in Appendix A. STARS has made significant progress as a statewide information system for the articulation of degree programs and transfer of academic credit over the past few years. Although the system has been in place for almost a decade, its role as a statewide information system in support of the general studies curriculum and the AGSC was achieved in the summer of 1998. Data in Table 1 show the use of the system for the past three years by students, advisors, faculty, administrators, and others. The increased use by students and advisors over the past two years is evident. The line graph in Figure 1 showing usage over the past 16 months illustrates that students and advisors are using the system. 3
Table 1. STARS Usage by Academic Year Students Counselors/Advisors Faculty Administrators Other Totals 45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 1998-1999 1999-2000 2000-2001 4
Figure 1. Usage Overview from Nov. 2000 - Feb. 2002 Students Counselors/Advisors Faculty Administrators Other Total 7000 6000 5000 4000 3000 2000 1000 0 November 2000 December 2000 January 2001 February 2001 March 2001 April 2001 May 2001 June 2001 July 2001 August 2001 September 2001 October 2001 November 2001 December 2001 January 2002 February 2002 5
Table 2. STARS Usage Totals by Two-Year Institution For the Time Period of July 2001 through December 2001 Totals 3000 2500 2000 1500 1,815 1,730 1,260 1,140 1,575 1,206 2,188 1,420 2,503 2,268 2,037 1,524 1000 500 0 450 Alabama Southern Community College Bevill State Community College 604 Bishop State Community College Calhoun Community College Central Alabama Community College Chattahoochee Valley Community College 794 Enterprise State Junior College Faulkner State Community College Gadsden State Community College 38 Harry M. Ayers State Technical College 380 Jefferson Davis Community College Jefferson State Community College 216 Lawson State Community College 728 Lurleen B Wallace State Junior College Northeast Alabama State Community College 562 Northwest-Shoals Community College Shelton State Community College Snead State Community College Southern Union State Community College 63 State ADPE Technical Colleges Wallace Community College-Dothan 769 Wallace Community College-Hanceville 505 Wallace Community College-Selma 6
Peak usage occurs during the spring semester pre-registration period in November each year and during registration periods in January and August each year. However, the data in Table 2 suggest that a wide degree of variation exists among community colleges in the use of STARS. Obviously some differences are attributable to the size of enrollment; however, some institutions with half the enrollment of another show usage data double that of the larger institution. Evaluation from the viewpoint of the users of STARS and from those who are involved in the articulation and transfer process could be useful in assessing the impact of Act 94-202, the role of the AGSC, and improving the STARS system. Rationale for the AGSC/STARS Evaluation Project June 2001 marked the completion of the seventh year of the AGSC and the third year of STARS as a statewide information system for facilitating the articulation of degree program requirements and the transfer of academic credit. While the accomplishments over this period have been remarkable, members of the committee believed it was time to assess how the AGSC and STARS were doing and to seek information for improving the quality and effectiveness of the operations and services. Committee concerns involved the degree to which the intent of Act 94-202 had been achieved, the effectiveness of the organizational structure of AGSC, how the AGSC was fulfilling its responsibilities, and the overall impact of the general studies curriculum and the articulation and transfer process on higher education in Alabama. The AGSC contracted with Dr. W. Donald Crump, former Associate Vice President for Academic Affairs and Professor of Education at The University of Alabama, to design and implement an evaluation of the general studies curriculum and STARS. Dr. Crump served as Project Coordinator. Dr. Marcia O Neal of The University of Alabama at Birmingham Center for Educational Accountability and Dr. Peggy Wilds of Higher Education Direction, a consulting firm in Kentucky, served as evaluators in the project. The concerns mentioned previously and the following two key questions guided the goals, design, and methods of the AGSC/STARS Evaluation Project: How are we doing? Where do we go from here? AGSC/STARS Evaluation Project Goals, Design, and Methods The Articulation and General Studies Committee identified four goals to give direction to the design and methods to be employed in the evaluation project. These goals were: 1. To determine the degree to which the intent of Act 94-202 has been achieved; 2. To evaluate the effectiveness of the organizational structure established by the statute; 3. To assess the degree to which AGSC is fulfilling its responsibilities; and 4. To evaluate the impact of the articulation process on higher education in Alabama. 7
The charge given by the AGSC emphasized the importance of addressing the two major questions and the four goals of the evaluation project from multiple perspectives. The evaluation was designed to include data and information from a variety of sources with high priority given to those individuals who are users of the STARS system, namely students and those community college and university personnel involved in advising students in the transfer process. Feedback from college and university presidents and chief academic officers, former members of the AGSC, and from legislators who were involved in the passage of Act 94-202 also was a component of the evaluation. These groups comprised the key constituency groups to be included in the assessment process. The evaluation design incorporated both quantitative and qualitative methods to address both the major questions and goals of the study. The primary strategies for evaluation included: 1. Questionnaire surveys to students, to community college and university advisors, faculty, administrators, and others, and to college and university transfer contact people; 2. Focus group discussions with community college students, university transfer students, community college advisors, and university transfer appeals officials; 3. Interviews with a sample of college/university presidents or chief academic officers, former AGSC members, and legislators; and 4. Data from the STARS system. The following sections provide additional information about these evaluation strategies, the methods employed, and those responding or participating in the evaluation. Questionnaire Surveys The questionnaire surveys were designed primarily to address the major question of how are we doing and to provide guidance for the focus group discussions. The questionnaires were designed, administered, and the results compiled by Dr. Marcia O Neal at the University of Alabama in Birmingham Center for Educational Accountability. Each of the three surveys included questions to provide descriptive information about the background and experience of the respondent and a range of items to solicit the respondent s ratings of features of the STARS system and the usefulness of the information. Copies of the questionnaires used for the survey of 1) students, 2) advisors, faculty, administrators, and others, and 3) transfer contact people are provided in Appendix B. For the student sample, surveys were mailed to approximately 1,500 students randomly selected from among the 30,760 who accessed STARS during the period from August 17, 1998, through July 3, 2001. Responses were received from 136 students and 43% of this number had transferred from a community college to a university. Among the respondents, four community colleges accounted for 40% of the responses to the question of where were you attending when you last used the STARS web site. Additional information about the respondents is shown in Appendix C. For the sample of advisors, faculty, administrators, and others, approximately 500 surveys were mailed to those randomly selected from the 1,749 individuals who had accessed STARS during the same period used in selecting students. Responses were received from 115 individuals and over 85% indicated they were employed at a community college. Additional information about these respondents is provided in Appendix D. Questionnaires were sent to the Transfer Appeals Contact Person at 19 community colleges and each of the 17 public universities in Alabama. Responses were received from 12 of the community colleges and 7 of the universities. Additional information is provided in Appendix E. 8
Focus Groups Focus group discussions were a crucial component of the evaluation project and were designed primarily to address the major question of where do we go from here. Focus groups are considered to be a qualitative evaluation strategy, less structured and more exploratory than quantitative methods. They allow evaluators to explore subjective judgments, to capture more detail, to gain a greater understanding of the perceptions, feelings, and attitudes expressed, and to evaluate the degree of consensus among participants. Results are not easily quantifiable, but can provide the basis for valuable insights and suggestions. The primary targets for the use of focus group discussions were those individuals directly involved in the use of the STARS system. These groups were: community college students; university transfer students; community college advisors/administrators; and university transfer appeals officials. The first step in the focus group phase of the evaluation was a preliminary review of the survey results. Following this, objectives for each of the focus groups were developed (see Table 3) and discussion guides were prepared in response to these objectives. Key contact people for the focus groups were identified, a schedule developed, and participants were invited. Nine focus groups were held between October 10, 2001, and February 7, 2002. A complete listing of the focus group meetings is available in Appendix F. Audiotape recordings and notes were taken at each session and comments were compiled for each of the four groups listed above. Dr. Peggy Wilds of Higher Education Direction, a consulting firm in Kentucky, organized the focus groups, conducted the discussions, compiled the comments, analyzed the results, and submitted a comprehensive report to the project coordinator. Dr. Wilds prepared the compilation of comments from each of the four groups listed above and developed the key insights and suggestions for improvements from each group. These will be reported in the section on evaluation results. In-Person and Telephone Interviews The in-person and telephone interviews with presidents or chief academic officers, former AGSC members, and legislators were designed to provide general information on the two major questions of how are we doing and where do we go from here. The primary emphasis of this component, however, was to evaluate the degree to which the intent of the legislation has been achieved, to evaluate the organizational structure of the AGSC and STARS, to assess the degree to which AGSC is fulfilling its responsibilities, and to assess the overall impact of the general studies curriculum and STARS on higher education in Alabama. Similar to the focus group discussions, the interviews employed a qualitative strategy exploring the perceptions, attitudes, and subjective judgments of the participants. Questions and discussions were based on the four goals of the evaluation project. Interviews were not audiotape recorded, but notes were taken during the interviews and then compiled and grouped by the Project Coordinator. The sample of presidents or chief academic officers included seven from universities and nine from community colleges and seven interviews were in-person and nine were by phone. For former AGSC committee members, the sample was seven with two in-person interviews and five telephone interviews. For the sample of legislators, all five individuals were sponsors of Act 94-202 and two were members of the Senate and three were House members. One interview was in person and four were by phone. 9
Table 3. Objectives for Focus Groups Discussions Community College Students Determine how the students learned about the STARS system. Assess how easily the students found the system to use. Identify the difficulties in using and preparing the Transfer Guide. Evaluate how the STARS system influenced the choice of transfer to a four-year college or university. Discuss how the STARS system influenced the choice of a major. Estimate the number of university web sites viewed by students. Delineate any difficulties in transferring academic courses for credit. Solicit suggestions for improving the STARS system. Community College Advisors/Counselors/Others Determine how the STARS system is used to advise students. Discuss the perceptions about the accuracy of information found on the STARS system. Evaluate how the STARS system has helped in advising students in choosing a major. Assess how the STARS system has helped in advising students in choosing a four-year college or university. Delineate the major advantages in using the STARS system. Identify significant problems encountered in using the STARS system. Solicit suggestions for improving the STARS system. University Transfer Students Determine how the students learned about the STARS system. Assess how easily the students found the STARS system to use. Identify the difficulties in using and preparing the Transfer Guide. Evaluate how the STARS system influenced the choice of transfer to a four-year institution. Discuss how the STARS system influenced the choice of a major. Estimate the number of college/university web sites that were viewed prior to choosing a major and/or college for transfer. Delineate difficulties in transferring academic courses for credit. Solicit suggestions for improving the STARS system. University Transfer Appeals Persons Determine how the STARS system has affected the overall transfer process at four-year colleges and universities. Identify concerns/problems encountered in getting accurate information displayed on the STARS system. Delineate concerns/problems encountered in maintaining and/or updating information on the four-year college or university website. Determine the use made of monthly reports from the STARS system. Evaluate the overall impact of the STARS system on four-year colleges and universities. Solicit suggestions for improving the STARS system. 10
Results for the AGSC/STARS Evaluation Project The evaluation design described in the previous section addressed the key questions and goals from multiple perspectives and employed both qualitative and quantitative strategies. The following sections report results from each of the key constituency groups. Results from Surveys of Students Evaluation results for the STARS system from the 136 students who responded to the survey were analyzed in several ways. First, results for each of the 32 items in the questionnaire for the total group of respondents were tabulated and the comments offered by 41 of the respondents were compiled. Subsequently, comparisons were made between and among groups of respondents categorized according to (1) transfer or non-transfer status, (2) year of last use of the STARS web site, (3) frequency of use of the web site, and (4) number of STARS guides requested. These comparisons were conducted to determine if any of these four variables might have been factors contributing to differences in the questionnaire results. The following sections report the results for the student survey and the related comparisons. Survey results for the total group of respondents are shown in Table 4. Over 80% of the respondents indicated Strongly Agree or Agree to 22 of the 32 items on the questionnaire. Eliminating those items that 15% or more of the respondents considered Not Applicable, 22 of the 24 remaining items were rated as Strongly Agree or Agree by 80% or more of the respondents. These results suggest that students generally are able to access the web site, transfer guides, college home pages, college transfer assistance pages, and college course listings and move around within these sources without difficulty; in addition, the information in these sources generally is considered complete, accurate, and useful by the respondents. Although many respondents indicated Not Applicable, locating Area V pages seemed a concern for a higher percentage of students. STARS information was considered helpful in making the decision to transfer and in selecting which university to attend by 60% or more of the students and over 55% agreed or strongly agreed that STARS information helped them make the decision about which major to choose. Over 75% indicated that STARS information made it easier for them to transfer and more than 80% indicated that STARS information helped them avoid taking courses that would not transfer. These results indicate an overwhelmingly positive response by students to the STARS system and its role in facilitating the articulation and transfer of academic credit between community colleges and state universities in Alabama. Table 4. Student Survey Results for Total Group (N-136) Opinion Item (Number Responding to Item) 1. I found the STARS web site without difficulty. (136) 2. I moved around the STARS web site without difficulty. (135) Strongly Agree Response Choices Agree Disagree Strongly Disagree Not Applicable Item Statistics 95% Confidence Interval N M SD LL UL 40% 44% 7% 3% 7% 127 3.29.74 3.16 3.42 40% 51% 5% 4% 130 3.36.58 3.26 3.46 11
Table 4. Student Survey Results for Total Group (continued) (N-136) 3. I located college transfer guide(s) I wanted without difficulty. (135) 4. Information in the college major transfer guide(s) was complete. (134) 5. Information in the college major transfer guide(s) was accurate. (135) 6. Information in the college major transfer guide(s) agreed with information from other sources. (131) 7. Information in the college major transfer guide(s) was useful to me. (134) 8. I located college home page(s) I wanted without difficulty. (135) 9. Information on the college home page(s) was complete. (135) 10. Information on the college home page(s) was accurate. (134) 11. Information on the college home page(s) agreed with information from other sources. (134) 12. Information on the college home page(s) was useful to me. (135) 13. I located college transfer assistance page(s) I wanted without difficulty. (134) 14. Information on the college transfer assistance page(s) was complete. (134) 43% 47% 5% 1% 4% 130 3.36.66 3.25 3.47 42% 50% 7% 1% 134 3.33.65 3.22 3.44 39% 51% 5% 3% 2% 132 3.28.70 3.16 3.40 33% 50% 8% 1% 8% 121 3.25.65 3.13 3.37 48% 48% 3% 1% 134 3.42.63 3.31 3.53 42% 47% 4% 6% 127 3.40.58 3.30 3.50 35% 53% 7% 1% 4% 129 3.28.62 3.17 3.39 37% 54% 3% 1% 5% 127 3.34.58 3.24 3.44 33% 48% 6% 1% 13% 117 3.29.63 3.18 3.40 39% 51% 5% 4% 129 3.36.58 3.26 3.46 30% 54% 6% 1% 8% 123 3.23.64 3.12 3.34 32% 53% 6% 9% 122 3.29.58 3.19 3.39 12
Table 4. Student Survey Results for Total Group (continued) (N-136) 15. Information on the college transfer assistance page(s) was accurate. (133) 16. Information on the college transfer assistance page(s) agreed with information from other sources. (133) 17. Information on the college transfer assistance page(s) was useful to me. (134) 18. I located college Area V page(s) I wanted without difficulty. (130) 19. Information on the college Area V page(s) was complete. (131) 20. Information on the college Area V page(s) was accurate. (131) 21. Information on the college Area V page(s) agreed with information from other sources. (131) 22. Information on the college Area V page(s) was useful to me. (131) 23. I located college approved course listing(s) I wanted without difficulty. (134) 24. Information on the college approved course listing(s) was complete. (134) 25. Information on the college approved course listing(s) was accurate. (133) 26. Information on the college approved course listing(s) agreed with information from other sources. (131) 32% 54% 5% 9% 121 3.29.57 3.19 3.39 31% 49% 8% 13% 116 3.27.61 3.16 3.38 35% 52% 3% 1% 9% 122 3.34.58 3.24 3.44 28% 45% 14% 14% 112 3.16.68 3.03 3.29 30% 45% 9% 16% 110 3.25.64 3.13 3.37 29% 47% 8% 16% 110 3.25.61 3.14 3.36 26% 46% 8% 20% 105 3.22.62 3.10 3.34 32% 45% 7% 1% 15% 111 3.28.65 3.16 3.40 39% 46% 7% 2% 5% 127 3.28.71 3.16 3.40 35% 49% 8% 1% 6% 126 3.25.68 3.13 3.37 36% 48% 8% 2% 6% 125 3.26.71 3.14 3.38 33% 46% 8% 2% 11% 116 3.24.69 3.11 3.37 13
Table 4. Student Survey Results for Total Group (continued) (N-136) 27. Information on the college approved course listing(s) was useful to me. (134) 28. STARS information helped me make the decision to transfer to a four-year school. (133) 29. STARS information helped me make the decision about which four-year school to attend. (133) 30. STARS information helped me make the decision about which major to choose. (132) 31. STARS information made it easier for me to transfer to a four-year school. (133) 32. STARS information helped me avoid taking courses that would not transfer. (135) 40% 48% 5% 2% 5% 127 3.31.69 3.19 3.43 29% 31% 16% 5% 20% 107 3.06.89 2.89 3.23 23% 32% 17% 6% 23% 103 2.91.91 2.73 3.09 24% 33% 20% 8% 16% 111 2.87.94 2.70 3.04 38% 37% 7% 2% 17% 111 3.32.74 3.18 3.46 45% 38% 8% 2% 7% 126 3.35.74 3.22 3.48 Comments from students responding to the survey reinforced the results and, in some cases, identified concerns and offered suggestions for consideration. A majority of the comments were positive (i.e., 25 of the 47) and included remarks such as STARS is a transfer student s dream come true and STARS is a great website and I would recommend it to anyone. Additional comments related to inconsistent or inaccurate information, website format and clarity, Area V pages, and similar topics. A compilation of the comments is provided in Appendix G. Comparisons related to the four factors described previously involved comparing the means and standard deviations of the ratings for each of the 32 items according to groupings appropriate to each of the variables. Ratings were calculated using a scale of Strongly Agree equals 4, Agree equals 3, Disagree equals 2, and Strongly Disagree equals 1. A rating of Not Applicable was not included in calculating the means and standard deviations. The comparison for students who have transferred from community colleges to universities to those who have not transferred indicated only 1 item among the 32 with a noticeable difference in the mean rating (See Appendix H, Table H-1). Item #18, which addresses locating Area V pages without difficulty, yielded an average rating of 2.98 for students who have transferred compared to 3.28 for those who have not, a value slightly less than Agree compared to a value slightly greater than Agree. Perhaps those students who have transferred may have accessed the STARS web site before Area V information was complete for many universities. The comparison of results based on year of last use of the STARS system did not indicate any differences as great as.30 or more among the three means (See Appendix H, Table H-2). However, a 14
cursory review of the ratings suggests a tendency for ratings by those who last used the system in 1998-99 to be lower than those who last used the system in 2000 and in 2001. The comparison based on frequency of use of the STARS system also indicated little difference in the mean ratings (See Appendix H, Table H-3). Only 1 item (#28) related to information that was helpful in deciding to transfer was rated noticeably lower (i.e.,.24). Apparently those who use the system only once already have decided to transfer and need less assistance in evaluating their options. Finally, the comparison based on the number of transfer guides requested or printed showed little systematic differences in the mean ratings between the two groups. Two items, #4 concerned with completeness of information in the college transfer guide and #30 concerned with usefulness of information in deciding on a major, showed some difference (See Appendix H, Table H-4). Based on these comparisons, overall results of the survey for students were not affected markedly by any of the four variables involving (1) transfer or non-transfer status, (2) year of last use of the STARS web site, (3) frequency of use of the web site, and (4) number of STARS guides requested. Some tendency existed for higher ratings among those who have last used the system in 2000 and 2001. Results from Surveys of Advisors, Faculty, Administrators, and Others Evaluation results for the STARS system from the 115 respondents for the survey of advisors, faculty, administrators, and others were analyzed in a manner similar to that used for the student survey. Results for each of the 44 items in the questionnaire were tabulated and the means, standard deviations, and 95% confidence intervals were calculated for each item based on a 4-point scale ranging from 4 for Strongly Agree to 1 for Strongly Disagree. Responses indicating Not Applicable were not included in calculating the means, standard deviations, and confidence intervals. Written comments offered by 42 of the respondents were compiled. Next, comparisons of the averages of ratings were made among groups of respondents categorized by (1) primary responsibility (e.g., advisor, faculty), (2) percentage of time spent advising, (3) frequency of use of STARS, and (4) number of students advised. The following sections report the results of the survey and the comparisons described previously. The survey results for all respondents are presented in Table 5 and were similar in some ways to the survey results for students. Approximately 80% of the respondents indicated Strongly Agree or Agree to 25 of the 44 items on the questionnaire. These results suggest advisors, faculty, administrators, and others using STARS are able to locate the web site, transfer guides, college home pages, college transfer assistance pages, and college course listings and to move around within the sources without difficulty. Opinion Item (Number Responding to Item) 1. I found the STARS web site without difficulty. (115) 2. I am able to move around the STARS web site without difficulty. (113) Table 5. Survey Results for Total Group of Advisors, Faculty, Administrators and Others (N-115) Strongly Agree Agree Response Choices Disagree Strongly Disagree Not Applicable Item Statistics 95% Confidence Interval N M SD LL UL 60% 37% 3% 115 3.57.55 3.47 3.67 43% 50% 5% 2% 113 3.35.67 3.23 3.47 15
Table 5. Survey Results for Total Group of Advisors, Faculty, Administrators and Others (continued) (N-115) 3. I can locate college transfer guide(s) I want without difficulty. (113) 4. Information in the college major transfer guide(s) is complete. (114) 5. Information in the college major transfer guide(s) is accurate. (111) 6. Information in the college major transfer guide(s) agrees with information from other sources. (111) 7. Information in the college major transfer guide(s) is useful to me. (115) 8. Information in the college major transfer guide(s) is useful to students. (113) 9. I can locate college home page(s) I want without difficulty. (114) 10. Information on the college home page(s) is complete. (114) 11. Information on the college home page(s) is accurate. (114) 12. Information on the college home page(s) agrees with information from other sources. (111) 13. Information on the college home page(s) is useful to me. (111) 35% 57% 8% 1% 113 3.25.63 3.13 3.37 14% 52% 24% 4% 6% 107 2.80.75 2.66 2.94 15% 65% 12% 8% 102 3.04.54 2.94 3.14 13% 59% 20% 2% 8% 103 2.88.65 2.75 3.01 43% 55% 1% 2% 113 3.42.51 3.33 3.51 47% 50% 1% 2% 111 3.47.52 3.37 3.57 35% 55% 4% 5% 108 3.32.56 3.21 3.43 13% 52% 23% 3% 10% 103 2.83.70 2.69 2.97 13% 62% 11% 2% 11% 101 2.98.60 2.86 3.10 12% 57% 18% 1% 13% 97 2.91.61 2.79 3.03 25% 66% 3% 6% 104 3.24.49 3.15 3.33 16
Table 5. Survey Results for Total Group of Advisors, Faculty, Administrators and Others (continued) (N-115) 14. Information on the college home page(s) is useful to students. (112) 15. I can locate college transfer assistance page(s) I want without difficulty. (114) 16. Information on the college transfer assistance page(s) is complete. (112) 17. Information on the college transfer assistance page(s) is accurate. (112) 18. Information on the college transfer assistance page(s) agrees with information from other sources. (112) 19. Information on the college transfer assistance page(s) is useful to me. (112) 20. Information on the college transfer assistance page(s) is useful to students. (113) 21. I can locate college Area V page(s) I want without difficulty. (111) 22. Information on the college Area V page(s) is complete. (111) 23. Information on the college Area V page(s) is accurate. (110) 25% 66% 2% 7% 104 3.25.48 3.16 3.34 20% 56% 16% 2% 6% 107 3.01.68 2.88 3.14 13% 53% 21% 3% 11% 100 2.86.70 2.72 3.00 14% 59% 10% 2% 15% 95 3.01.61 2.89 3.13 13% 54% 15% 2% 15% 95 2.94.65 2.81 3.07 25% 63% 1% 12% 99 3.27.47 3.18 3.36 28% 60% 2% 10% 102 3.29.50 3.19 3.39 26% 42% 22% 5% 5% 105 2.95.84 2.79 3.11 11% 44% 29% 8% 8% 102 2.63.81 2.47 2.79 11% 53% 19% 6% 11% 98 2.77.76 2.62 2.92 17
Table 5. Survey Results for Total Group of Advisors, Faculty, Administrators and Others (continued) (N-115) 24. Information on the college Area V page(s) agrees with information from other sources. (110) 25. Information on the college Area V page(s) is useful to me. (110) 26. Information on the college Area V page(s) is useful to students. (111) 27. I can locate college approved course listing(s) I want without difficulty. (111) 28. Information on the college approved course listing(s) is complete. (111) 29. Information on the college approved course listing(s) is accurate. (110) 30. Information on the college approved course listing(s) agrees with information from other sources. (109) 31. Information on the college approved course listing(s) is useful to me. (112) 32. Information on the college approved course listing(s) is useful to students. (107) 33. STARS information helps students make the decision to transfer to a four-year school. (113) 9% 51% 24% 5% 11% 98 2.71.73 2.57 2.85 25% 58% 8% 1% 8% 101 3.16.61 3.04 3.28 28% 56% 6% 1% 9% 101 3.22.61 3.10 3.34 26% 62% 9% 3% 108 3.18.58 3.07 3.29 16% 65% 8% 2% 9% 101 3.05.59 2.93 3.17 17% 64% 8% 1% 10% 99 3.08.57 2.97 3.19 15% 62% 9% 2% 12% 96 3.02.60 2.90 3.14 29% 65% 1% 5% 106 3.29.48 3.20 3.38 29% 63% 2% 7% 100 3.29.50 3.19 3.39 33% 48% 12% 8% 104 3.23.66 3.10 3.36 18