2012 Enhancing Research and Institutional Effectiveness San José Evergreen Community College District DRAFT 5/18/2012 Research and Institutional Effectiveness RIE
The district, in collaboration with San Jose City College and Evergreen Valley College, should develop and implement a plan that will provide the necessary resources to increase college and district capacity to analyze progress towards achieving institutional goals and objectives including improvements in student achievement and student learning. (IB.2, IB.3, IB.4, IB.5, IB.6, IIA.2c, IIA.2e, IVB.3b, IVB.3c) In response to the above referenced shared recommendation of the Accrediting Commission of Community and Junior Colleges, the Office of Research and Institutional Effectiveness has developed a plan for increasing research capacity at the San Jose/Evergreen Community College District. The plan is designed to address the following priorities: 1. Define the roles and responsibilities 2. Outline the critical needs for research at the colleges and district office 3. Assign resources/responsibilities to each critical need 4. Comprehensive approach to mandated reporting 5. Timeline for implementation of on-going assessment 6. Plan and timeline for measuring institutional goals and targets 7. Comprehensive Institutional Effectiveness Plan Page 1 of 14
Roles and Responsibilities San Jose/Evergreen Community College District Mission With student learning as its primary focus, the San José/Evergreen Community College District provides continuous access to a wide range of comprehensive and flexible post-secondary academic and occupational programs that prepare Silicon Valley residents of all ages for balanced and productive lives and successful careers. To reach these educational goals, the district provides the appropriate support services to meet the needs of an increasingly diverse student population. In fulfilling this mission, the district assists students in achieving their educational and employment goals including, but not limited to, lower-division transfer and general education, two-year college degrees, certificates, distance learning opportunities, basic skills instruction, English as a Second Language, economic development, adult non-credit courses, targeted career training, and community services courses. This mission is, and shall remain, in accordance with all legislation enacted by the State of California. The mission of the Office of Research and Institutional Effectiveness (ORIE) is to support the mission, vision, and values of the San Jose Evergreen Community College District through establishing, implementing, and monitoring cycles of continuous quality improvement at the district level, at the workforce institute, and at each college. The mission of the ORIE is realized through the pursuit of ongoing Operational Objectives that are evaluated on an annual basis: a. Develop and Implement an Institutional Effectiveness Plan b. Develop a district-wide standard for program review c. Monitor and analyze strategic performance indicators for the district, the workforce institute, and the college d. Develop a long-range district-level planning process e. Develop a Scorecard for monitoring district-level performance indicators f. Maintain accurate and up-to-date information as communicated through the ORIE website g. Develop a timeline for implementation and plan for on-going assessment of district-level objectives h. Conduct ongoing research into matriculation, student success and program effectiveness for use in decision-making Page 2 of 14
i. Provide ongoing education to the district-wide community about assessment, evaluation, measurement and program review j. Monitor and guide mandated curriculum planning/reporting for the colleges and the workforce institute k. Explain the procedures for monitoring the development of courses and approving programs l. Provide a structure for consistent documentation of the content and objectives of programs and courses The role of research and institutional effectiveness is to support the district and colleges in monitoring programs and processes with the goal of making decisions for continuous quality improvement and building a culture of evidence. The functional delineation of roles and responsibilities makes it clear what the responsibility of the district office of research and institutional effectiveness is in comparison to that of the colleges in forwarding the research function. The relationship between the district office and the colleges with regard to the research function in symbiotic. Without support from the other, neither the colleges nor the district office would adequately fulfill their responsibilities. Roles and Responsibilities for Institutional Effectiveness Functions Responsibilities/Role Function District RIE Office Colleges Student Outcomes Institutional Outcomes Curriculum Management Disseminate common definitions and metrics; provide templates for reporting; Monitor progress of colleges and district metrics; continuing professional development to the colleges Update scorecard and communicate progress to internal and external communities; Maintain record of college scorecard and provide support to colleges as needed Provide information and updates on state and federal regulations/title V Education Code; support the flow of curriculum process; support Identify, maintain, and measure student outcomes for all courses and program; identify, maintain, and measure general education outcomes; report to district annually about progress on student outcomes Update outcomes on an annual basis for college s strategic performance indicators. Report information to district office annually. Update curriculum and programs according to content standards and student/community needs; implement curriculum review Page 3 of 14
Responsibilities/Role Function District RIE Office Colleges update of curriculum handbook; edit updated course descriptions each semester for course schedules and annually for course catalogues; coordinate and monitor technical aspect of course and section setup in Datatel; Final point of activation for course schedule in Datatel; review program guides in Datatel for accuracy and functionality and changes according to the curriculum flow process; update course descriptions for the course schedules and catalogues; input course sections and schedules into Datatel; determine prerequisites and course requirements for programs; build program guides into Datatel s Degree Audit Mandated Reporting Monitor submissions of required Program Review Evaluation of Governance Systems Strategic Planning State databases by IT; Provide data to colleges and/or functional areas. Facilitate update of data portal; assist colleges and areas in interpreting program review data; maintain standardized program review templates for use by both colleges and district office. Periodic survey and analysis of district-wide governance systems Provide periodic environmental scanning reports; analysis of performance indicators; develop, maintain and explain the process for cyclical planning and assessment of goals and objectives Develop program review model for college in conjunction with Academic Senate; monitor process through college-wide committees; use data to make decisions about program status Periodic survey and analysis of college-level governance systems Periodically review college level mission, vision, and strategic objectives; develop annual goals and objectives; develop performance indicators for college level strategic objectives; monitor progress of strategic goals Page 4 of 14
Critical Needs for Research and Institutional Effectiveness Assessing Critical Needs Building on the structure of reporting and data-mining by the office of Research and Institutional Effectiveness (RIE) in collaboration with Information Technology Support Services (ITSS) referenced in the follow-up report of October 15, the district in conjunction with the colleges has continued to enhance the research services available. As a method for assessing the needs of the district, colleges, and workforce institute, a district-wide survey was conducted. Faculty and staff at the district office, San Jose City College, Evergreen Valley College, and the Workforce Institute were asked how often they had used the services of the Office of Research and Institutional Effectiveness in the past 12 months. Those who had received services from RIE were asked to rate the quality of those services from a variety of perspectives. All employees were asked their opinion on how well RIE had performed functions critical to the offices mission and objectives. Of the 218 responses, 72% (158) had never asked for services from RIE within the past 12 months. Of those who had asked for services, customer service ratings were moderate at best. Those who had asked for services from the RIE department indicated that the staff is helpful and courteous, but just over half of the respondents thought positively about the follow-through of staff, clarity and consistency of procedures, and if service provided met their needs. Percent of RIE customers rating the quality of service as Excellent or Good Service component Positive rating Helpfulness of Staff 66.0% Courtesy of Staff 71.2% Competency of Staff 63.8% Follow-through of Staff 58.6% Accuracy of Information Provided 64.4% Service met my needs 59.6% Timeliness of response 60.3% Clarity and consistency of procedures 51.4% Opportunity to provide input 57.9% Overall quality of service 62.1% Responses to the statements related to the RIE mission and objectives show that most of the respondents to the survey do not know what RIE does or do not believe they need research services. Page 5 of 14
Percent of district faculty and staff who agree or don t know statements related to RIE functions Statement Positive rating Don t know Information received from RIE over the past 12 17.3% 59.1% months has met the needs of my department I understand the procedures for monitoring the 17.5% 54.6% development of courses and approving programs. The support provided by RIE for curriculum 15.6% 58.0% development is helpful. Data provided from the RIE office over the past 12 17.3% 63.0% months are accurate. In would rather gather information and data myself than ask the RIE office for support.* 10.2% 58.2% *This stem is backwardly coded -- disagree or strongly disagree is positive These survey results indicate a strong need to address the presence, availability, and capacity of RIE services throughout the district. The critical needs fall into four major areas: data mining; documenting progress toward achieving institutional goals; and expand the capacity of research and institutional effectiveness at the district and at the colleges. The current plan is designed to address the timeline for implementing processes for sustainability in these areas. Data Mining Access to data for decision making In addition to successfully accessing data from the Reporting Portal, as well as obtaining data from other sources such as the California Community College Chancellor s Office Data Mart, the California Partnership for Achieving Student Success (CalPASS), and the Integrated Postsecondary Information System (IPEDS) data center, the office of Research and Institutional Effectiveness has accessed data directly from the data warehouse in order to create and analyze customized data for planning, program review, and ad hoc reporting. Currently, the data warehouse is populated with data from a number of student and course tables from Colleague and a small amount of data from faculty tables. The RIE office has recently installed tools used to mine data from the data warehouse. SQL Server Management Studio, a powerful data analysis tool, is installed on the RIE server as is Hyperion. Hyperion is a tool which extracts tables and delivers common data from the data warehouse to end-users via the web. RIE has implemented a training process for end-users from San Jose City College, Evergreen Valley College, and the Workforce Institute. The model includes a train-the-trainer aspect in which high-end power-users train other power-users, who in turn facilitate the training and access of users at their college or work-location. Below is the timeline for implementation of Hyperion training which is on track to be complete at the end of June, 2012. Page 6 of 14
Timeline for Deployment of Hyperion Data Management Solution Timeframe (general) Users Learning objectives Outcomes Preliminary Session Tamela Hawley Overview of Hyperion Identify the uses of the (2 hours) Dolly Zen platform software March 15 Determine the extent of Bob Barr, Tamela Hawley knowledge within Mega Power Users Session 1 (6 to 8 hours) March 20 March 26 Bob Barr Session 2 (6 to 8 hours) March 28 April 3 Bob Barr Work sessions April 9 April 20 Mega Power Users Session 3 (4 to 6 hours) April 20-30 Bob Barr Session 4 (4 to 6 hours) --several sessions May through June SJECCD Trainers Mega Power Users/Designers Mega Power Users/Designers Mega Power Users/Designers Power Users End Users Basics of the Hyperion Platform Elements of designing templates in Hyperion Navigate the Hyperion platform Import tables from data warehouse Join Tables Pivot Tables Manipulate data Create charts Create forms and templates Combine graphs and tables Create usable docs for data analysis Export data to web Study Session Discuss training for endusers/identify trainers Discuss tables we could create Delegate tasks Basics of Hyperion Platform/Basics of design Accessing and using data Open tables Join Tables Create basic charts Download tables from web Download tables from web Understand how to read and use data for program review Hyperion has been used to prepare a comprehensive Fact Book for each college as well as a data sheet to support Program Review for each academic department at both San Jose City College and Evergreen Valley College. As of April 2012, the Fact Book for each college is located on the SJECCD website under Research and Institutional Effectiveness, with a link to college websites. The Program Review Data Sheets are available on a server shared by the entire SJECCD community in a folder called Program Review. Since April 2012, when the data sheets were disseminated, they have been used by academic departments, grants managers, as well as finance departments at both colleges to inform decision-making and reporting requirements. Using the Hyperion tool, RIE staff will update the fact books and the program review data sheets annually after the census cut-off in the fall. In addition, the RIE department has installed the ArcView suite of software. This software enables the district to analyze data housed in the data warehouse and link it to geographic data derived from census and mapping data for the surrounding area. The installation of ArcView Page 7 of 14
enhances the capability of the research office to provide powerful and useful data to departments looking for data to use in program reviews as well as provide environmental scanning data for use in strategic and academic planning. The RIE office has implemented a research requesting process that can be accessed via the District s website. College and district end-users who need access to data or research that is not already available in the Fact Book or in the various databases publically available through the California Community College Chancellor s Office, can request a study through the use of the research request process. The process requires that requestors obtain approval from their area administrator (dean, vice president, or vice chancellor) and the RIE office works with the requestor to determine how the request can be fulfilled including timelines. An example of this process in action is the request from the Menthas at San Jose City College program for data related to Peer Led Team Leadership (PLTL) program. The RIE office generated a series of tables that compared the grades and course completion of students in LS 210 classes that included the PLTL model to those that did not. Results of this analysis was presented at the annual PLTL conference held on June 29, 2012. Documenting progress toward achieving institutional goals Using data to monitor institutional outcomes In May 2012, RIE created a template for monitoring institutional performance indicators designed to support the district and the colleges in assessment of these indicators on an annual basis. The monitoring tool uses the strategic performance indicators agreed upon through the planning structures of the district or the colleges as specified in their respective policies. Goals and targets are determined in conjunction with the executive teams for each unit and are based upon historic data where available, and peer groups defined by the California Community College Chancellors Office. At the end of each fiscal year, RIE will provide an update to the performance indicators using the most recent data. The data will be shared with the executive teams, governance committees, and stakeholders in the fall of each fiscal year. Using a common template forwards the district s ability to align the strategic objectives of the colleges to those of the district, enhancing the capability of sustaining an integrated research paradigm across the district. Below is a sample of the monitoring template for tracking strategic performance indicators for San Jose City College. Page 8 of 14
Sample performance monitoring tool San Jose City College College Strategic Goal #1: Promote Student Success Key Performance Indicator: 1A: Increase the persistence rate of first-time students each year. 2008 2009 2010 Target Difference All First-time Students 67% 65% 65% 70% 5% 1B: Increase the successful course completion rate of both full-time and part-time students each year. 2008 2009 2010 Target Difference Full-time Students 73% 76% 75% 80% 5% Part-time Students 49% 47% 51% 60% 9% Page 9 of 14
Measuring Student Success The Board of Trustees has established a district-wide study group charged with developing a common definition student success. The committees charge includes developing a strategy for measuring student success as defined by the committee. The work of the committee is integral to aligning the district and college s institutional performance measurement, particularly as it pertains to student outcomes. During Fiscal Year 2012-2013 the RIE office will standardize operational definitions of the measures agreed upon by the Student Success Task-force as well as formalize the sources of data, including student surveys that are designed to measure student engagement and critical thinking skills. This includes the Community College Survey of Student Engagement (CCSSE) which was used in 2010 by San Jose City College. Establish and Implement a district-wide Institutional Effectiveness Plan On June 11, 2012 the district office participated in a workshop designed to support each department in developing a program review plan. Each functional area came away with a mission statement and a set of objectives which they can use to measure their success. A districtwide survey was administered by the RIE office in spring 2012, designed to assess how effectively each of the district offices service the colleges and community in fulfilling their individual office missions. This survey will continue to be conducted on a biennial basis. In order to sustain these plans, the RIE office has developed a template for program review in functional areas in the district office. On an annual basis, the program reviews will be reported as a part of the cyclical institutional effectiveness process. A committee made up of representatives from each of the main executive functions (Human Resources, Fiscal Affairs, Research and Institutional Effectiveness, Information Technology, and Workforce Institute) at the district office will analyze program review reports and monitor the process of responding to opportunities for improvement as a result of the annual review cycles. The Institutional Effectiveness Plan for SJECCD is based upon a performance measurement cycle of Plan Do Check Act. The annual cycle includes the district and all units developing goals based on missions and objectives (Summer 2012), identifying performance measures with baselines and targets (Summer 2012), establishing timelines for implementation and assessment (Fall 2012), review of results of assessment (Spring 2013), and making decisions for refinement of processes (Spring 2013). This model is depicted in the figure below. The timeline for performance measurement begins at the district level with the analysis of performance metrics for the districts strategic objectives. Both colleges performance indicators, aligned with the district plans, are also measured and analyzed on an annual basis. The RIE office supports the performance measurement process by providing valid data on an annual basis for the purposes of updating the yearly performance metrics. Page 10 of 14
Performance Measurement Cycle Refine Processes Develop Goals Review Results PDCA Identify Performance Measures Establish Timelines for Tracking Performance Establish Baselines and Targets Expand the capacity of Research and Institutional Effectiveness Since January 2011, RIE has worked to strengthen its capacity to serve the need of the colleges and other district offices for accurate and timely data. The Reporting Portal continues to be populated with custom reports that provide information to end-users about Human Resources, Research, Instruction, Program Review, Enrollment Management, and Student Affairs. Accessibility has increased substantially as usage grows. Communication is strengthened through the continued growth of user-groups such as the district-wide Technology Leadership Group, the Power Users Group at San Jose City College, and the Institutional Effectiveness Committee at Evergreen Valley College. These groups have been essential in helping to identify areas of need in terms of data integrity and accessibility. This is evidenced in the agendas from these groups over the period since January 2011 (see Evidence --) The Office of Research and Institutional Effectiveness has continued to enhance its capacity to support the efforts of the district and the colleges to access valid and reliable data and processes to make data-driven decisions. Based on in-depth assessment of the status of research support within the district, it was determined that the key factors for expanding RIE focus on accessibility of the office expertise to the rest of the district community an enhancing the human and professional resources of the office. In the spring of 2012, a comprehensive assessment of the website was completed by the new Executive Director of Research and Institutional Effectiveness. As a result of assessment, the website was enhanced to include the posting of the colleges fact books, posting of up-dated Page 11 of 14
contact information, and an interactive Research Request form that enables anyone within the district to submit requests for data. The research request process is fully explained on the website and the turn-around for first contact is promised within 48 hours. From March through June 2012, the new Executive Director of Research and Institutional Effectiveness spent time assessing the research support needs of each of the college and the workforce institute through interviews and by attending meeting of key constituencies at each work location. As a result of this assessment, in June 2012, a temporary research analyst was hired to support the college requests for data and a full-time permanent research analyst will be hired within Fiscal Year 2012-2013 Currently, the human capacity in the RIE office has been enhanced to include a secretary, a research analyst, a curriculum coordinator, and an executive director. Over time, the goal is to have a half-time researcher assigned to each college, but linked to the RIE office. In this manner the colleges would have needed research support available on an intimate level while still leveraging the professional expertise and resources available at the district office. This model will be realized in Fiscal Year 2013-2014. Communications Model for Building RIE Support DO Institutional Effectiveness EVC Research Analyst SJCC Research Analyst DO Research Analyst Institutional Effectiveness Committee Power Users Group Activities related to enhancing research and institutional effectiveness The table below summarizes the activities that have or will be completed compared to ACCJC accreditation requirements. The timeline for completion of these activities is included in the appendix to this document. Page 12 of 14
Standard Requirement (abvr.) Activity IB.2 Sets goals; articulates goals; sets measurable objectives; communicated goals and work toward achievement Strengthen the District s Strategic Planning Model and facilitate strategic planning for the district IB.3 IB.4 IB.5 IB.6 IIA.2c IIA.2e IVB.3b IVB.3c Assesses progress towards stated goals; decisions based on systematic cycles of evaluation; integrated planning; resource allocation; implementation; re-evaluation; qualitative and quantitative data are analyzed to make decisions Provides evidence that planning process is broad-based; opportunities for input by appropriate constituencies; allocates resources; leads to improvement Documented assessment results communicated to constituents Assures effectiveness of planning and resource allocation process by systematically reviewing and modifying the cycle High-quality instruction breadth, depth, rigor, sequencing, time to completion, synthesis of learning Evaluates all course and programs through on-going systematic review; achievement of learning outcomes; future needs and plans District provides effective services that support colleges in their missions and functions District provides fair distribution of resources that are adequate to support operation of the colleges Developed and implemented a performance report card for the key strategic performance indicators at the district level. Implemented program review at the district level Continually monitoring program review at the college level Develop and implement an Institutional Effectiveness Model Each college has developed SLOs and program reviews and publicized the results of these reviews. The district RIE office keeps track of the progress in a central database. Strengthen the District s Strategic Planning Model to include budget development. Continue monitoring Program Review and SLOs Continue monitoring Program Review and SLOs Implement and monitor program reviews at the district level Strengthen the District s Strategic Planning Model to include budget development Page 13 of 14
Relationship of Research and Institutional Effectiveness to District and College Functions Academic Affairs Enrollment Management Data Program Review Data Training on SLOs Curriculum and Scheduling Chancellor's Office and Presidents' Offices Student Affairs Strategic Performance Indicators Mandated Reporting Institutional Effectiveness Accreditation Research and Institutional Effectiveness Enrollment Management Data Student Engagement Survey Market Share Statistics Enrollment Projections Facilities and Auxiliary Services Business and Finance Mandated Reporting Enrollment Projections Functional Area Assessment Human Resources Mandated Reporting Functional Area Assessment Mandated Reporting WSCH Functional Area Assessment