CGS Assessment Report: MBA Program 2015-2016 Table of Contents Assessment Process Overview... 2 Assessment Process... 4 Results: Direct Measures of Student Learning... 5 Results: Indirect Measures of Student Learning... 7 EBI Alumni Survey... 7 IVY Test... 9 Results: Key Performance Indicators... 9 Progress Report on 2015-2016 Continuous Improvement Action Plan... 10 Continue to Improve MBA Student Learning Efficiency in Finance In Progress... 10 Continue to Improve Instructional Quality in the MBA Program Completed... 10 To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum Completed... 11 To Prepare for the Quarters to Semesters Conversion In Progress... 11 To Meet New Outcomes Assessment Requirements of IACBE Completed... 11 2016-2017 Continuous Improvement Action Plan... 11 To Ensure Quality Transition from Quarters to Semesters... 12 Continue to Improve MBA Student Learning Efficiency in Finance... 12 To Refine the PO#3 Rubric... 12 To Solve the Potential IVY Test Issues... 12 To Evaluate the Impacts of New Marketing Firm and Marketing Strategies... 12 Appendix A. Center for Graduates Studies Institutional Outcomes... 13 Appendix B. MBA Program Outcomes... 14 1
Assessment Process Overview The Center for Graduate Studies embraces a philosophy of continuous quality improvement and requires program administrators to use a variety of robust assessments to ensure that the stated mission and goals are achieved. Both internal and external assessments are utilized to monitor and evaluate the graduate program, allocate resources, create professional development, and update processes as part of the continuous quality improvement cycle. Specifically, the graduate program assessment process is designed to evaluate data from three areas: 1) direct measures of student learning outcomes, 2) indirect measures and 3) key performance indicators. Faculty members, in collaboration with instructional designers, are responsible for developing standardized assessment materials to be used within courses. Authentic assessment materials are designed to evaluate student capabilities as they relate to program and institutional outcomes. These standardized assessment instruments become a part of the course, and all faculty members teaching the course are required to administer the instruments. It should be noted that all standardized assessment instruments are developed with the intent to embed the assessment process within the course. In this manner, students are not asked to complete additional assignments or assessments beyond those that are a part of the normal educational process. This embedding of assessment measures is important to the faculty of the Center for Graduate Studies, who believe that assessment should be an integral piece of the educational process, not an addition to it. The assessment materials are designed to support faculty members in their classroom assessment and evaluation, present students with clear expectations and performance parameters, and provide students with detailed feedback on performance as it relates to learning outcomes. In addition to the direct measures, data are collected through the use of indirect measures, including surveys of program graduates, employer surveys, and/or accrediting agency reports. These data are combined with direct measures to complete the assessment data set. Key Performance Indicators (KPI) have been developed to complete the assessment plan. These KPIs are intended to measure programs in relation to priorities that have been set by the Institution based on our mission and values. The Center has identified the following as KPIs for evaluating the success of our graduate programs: Enrollments Retention Graduation rates Employment rates of graduates Faculty credentials These KPIs provide data for analysis and evaluation on metrics beyond teaching and learning. These metrics provide the primary operational data necessary for evaluating the stability of the program as well as for planning, budgeting, high level assessment of operations, and how the 2
program contributes to the mission and guiding principles of the institution. Additionally, these metrics are compared across graduate programs developing benchmarks, internal targets, and minimum performance standards. Annually, the program administrator has the responsibility of compiling the data, discussing and analyzing the data with the Faculty Council, and collaboratively developing a continuous improvement plan. The continuous improvement plan is designed to identify the steps necessary for improving student learning in the designated areas. To address specific findings, the plan may include identifying actions such as redevelopment of a course, seeking additional data to clarify student achievement, or requesting an alteration of specific assignments or teaching strategies to improve attainment of learning outcomes. Based on the findings, the plan may also include operational alterations to such areas as student services or faculty development. In addition to a review of data collected, the program administrator and the Faculty Council will undertake an annual review of the program assessment plan to determine the effectiveness of the plan, and the quality and usefulness of the data collected. As a portion of this annual review, it is anticipated that the assessment plan for each program will remain a dynamic document, continuing to evolve as the faculty become more experienced in the process of program assessment. 3
Assessment Report Program: Master of Business Administration (MBA) Dean: Dr. Na Lina Li Year: 2015-2016 Assessment Process 1. Collect data regarding: a. Student learning (direct measures/course embedded assessments) b. Indirect measures (Alumni perceptions regarding learning, faculty, and curriculum) c. Key performance indicators 2. Review and analyze data with the following stakeholders: a. Graduate Faculty Council b. Advisory Board c. Center for Graduate Studies Administration 3. Develop a Continuous Improvement Action Plan in collaboration with faculty 4. Submit assessment report: a. Chief Academic Officer b. IACBE c. System Coordinator for Academic Assessment 5. Publish assessment report: a. Faculty consumption b. Student consumption c. Staff and other stakeholders 6. Implement Continuous Improvement Action Plan 7. Review progress on the Continuous Improvement Action Plan of the prior year assessment report 4
Results: Direct Measures of Student Learning The results of the direct measures of student learning, for both Center for Graduate Studies (CGS) institutional outcomes (Appendix A) and MBA Program outcomes (Appendix B), are demonstrated in Tables 1 & 2 and Figures 1 & 2. The CGS institutional outcomes assessment results are positive. Ninety percent (90%) of students or more performed at a satisfactory or higher level on all outcomes as shown in Table 1 and Figure 1. This is consistent with the previous years. Table 1. Student Performance on CGS Institutional Outcomes 2015-2016 Exemplary Accomplished Satisfactory Approaches expectations Below expectations Satisfactory or above IO1 90.0% 10.0% 0.0% 0.0% 0.0% 100.0% IO2 58.8% 24.4% 11.6% 3.7% 1.5% 94.8% IO3 56.7% 22.8% 11.9% 6.1% 2.6% 91.4% IO4 62.4% 27.7% 7.3% 1.8% 0.8% 97.4% IO5 92.3% 6.7% 0.2% 0.0% 0.6% 99.2% IO6 62.4% 27.7% 7.3% 1.8% 0.8% 97.4% Mean 70.5% 19.9% 6.4% 2.2% 1.1% 96.7% Figure 1. Student Performance on CGS Institutional Outcomes 2015-2016 Assessment results on MBA Program outcomes are shown in Table 2 and Figure 2. Ninety percent (90%) of students or more performed at a satisfactory or higher level on all MBA Program outcomes. Student performance on most outcomes were similar to last year. It is worth noting that more than 90% of students performed at a satisfactory or higher level on PO#11 (analyze the impact of information systems and technology on a business and 5
demonstrate the ability to make effective information management decisions). This is a second year in a row. In the 2013-2014 and prior academic years, less than 90% students reached the satisfactory or higher levels. Performance regarding this outcome will continue to be carefully monitored. On the other hand, the percentage of satisfactory or higher level performance on PO#3 (Collect, interpret and analyze existing and/or original research, using quantitative and statistical tools, and use in the decision making process) decreased from 98.4% to 90.0%. This does not reflect any drop in student performance. It occurs because the labels of levels were realigned in order to more accurately reflect the level descriptions. For example, previous Satisfactory is now labeled as Approaches expectations. The rubric that measures PO#3 will be refined in the 2016-2017 year. Table 2. Student Performance on MBA Program Outcomes 2015-2016 Exemplary Accomplished Satisfactory Approaches expectations Below expectations Satisfactory or above PO1 62.4% 27.7% 7.3% 1.8% 0.8% 97.4% PO2 92.3% 6.7% 0.2% 0.0% 0.6% 99.2% PO3 43.3% 30.0% 16.7% 7.8% 2.2% 90.0% PO4 80.8% 12.0% 6.2% 0.3% 0.6% 99.0% PO5 81.4% 14.7% 3.2% 0.6% 0.0% 99.4% PO6 80.8% 12.0% 6.2% 0.3% 0.6% 99.0% PO7 70.1% 15.6% 7.1% 4.3% 2.9% 92.8% PO8 33.9% 32.9% 28.5% 4.1% 0.6% 95.3% PO9 43.9% 35.6% 13.0% 4.8% 2.7% 92.4% PO10 90.0% 10.0% 0.0% 0.0% 0.0% 100.0% PO11 66.7% 15.8% 9.2% 5.5% 2.9% 91.0% PO12 28.6% 50.7% 14.3% 5.1% 1.3% 93.7% Mean 64.5% 22.0% 9.3% 2.9% 1.3% 95.8% 6
Figure 2. Student Performance on MBA Program Outcomes 2015-2016 Results: Indirect Measures of Student Learning EBI Alumni Survey The EBI Alumni Survey was again utilized to understand student perceptions of the MBA Program in the 2015-2016 academic year. Sixty-seven (67) graduates participated in the survey. Table 3 shows the results of this indirect measurement of the MBA Program outcomes. EBI provides external comparisons on PO2-8 and 11. Our graduates ratings were equal to or higher than the means of national participating institutions on all of these outcomes. No external comparison was provided on PO 1, 9 or 10. We do have preset expectations on these outcomes (i.e., >=5 on a 7-point Likert scale or <=2 on a reversed Likert scale). Our graduates perceptions on the three outcomes met the expectations. It is worth noting that student perception of their achievement on PO1 (Communicate using the advanced oral and written communication skills necessary for success in the business environment) increased from 4.93 in the 2014-2015 year to 5.00 this year. This may be a result of several endeavors. These included the implementation of several oral presentation opportunities in the curriculum, the development and increased use of rubrics that help 7
students to improve their writing skills, and the increased formative feedback opportunities provided in classes. However, readers should be cautioned that the increase was small and may be due to random variation. Measurement results of this PO will continue to be monitored. PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 PO11 Table 3. EBI Survey Results on MBA Program Outcomes (POs) Measured by Results Written and Oral Communication Skills questions Ethics and Legal Issues questions Data-Driven Decision-Making questions Domestic and Global Economies questions Critical Thinking and Problem Solving questions Critical Thinking and Problem Solving questions Financial Information questions Organizational Behaviors questions Multicultural and Diversity questions Professional Development questions Use and Manage Technology questions Average rating of Baker College MBA graduates (5.00) = 5.0 on a 7-point Likert scale. Average rating of Baker College MBA graduates (6.12) > national average of participating institutions (5.66). Average rating of Baker College MBA graduates (5.95) > national average of participating institutions (5.64). Average rating of Baker College MBA graduates (5.60) > national average of participating institutions (5.59). Average rating of Baker College MBA graduates (5.70) = national average of participating institutions (5.70). Average rating of Baker College MBA graduates (5.70) = national average of participating institutions (5.70). Average rating of Baker College MBA graduates (5.60) > national average of participating institutions (5.58). Average rating of Baker College MBA graduates (5.90) > national average of participating institutions (5.77). Average rating of Baker College MBA graduates (5.80) > 5.0 on a 7-point Likert scale. Average rating of Baker College MBA graduates (1.95) < 2.0 on a reversed 7-point Likert scale. Average rating of Baker College MBA graduates (5. 61) > national average of participating institutions (5.18). Figure 3 shows student ratings of the MBA Program in three overarching categories, student overall satisfaction, overall learning, and overall program effectiveness. All three measures continued to fall in the more satisfied area, with ratings of 6.11, 6.12, and 6.02 on 1-7 Likert scales. These ratings were slightly lower than last year. Despite this, all three ratings were 8
either significantly higher or had no difference than the selected six institutions and all participating institutions. Figure 3. Baker College MBA Program Overall Ratings EBI provides comparisons among institutions on 18 factors such as quality of instruction, breadth of curriculum, aspects of courses, etc. In the 2015-2016 academic year, our MBA program performed significantly better than the average of the selected six (6) institutions and the average of all participating institutions on Quality of Instruction, Learning Outcome on Ethical and Legal Responsibilities, Learning Outcome on Data-Driven Decision-Making, and Learning Outcome on Use and Manage Technology. We also performed significantly higher than the selected six (6) institutions on Overall Learning. Our performance on the other factors was not statistically different than the other institutions. IVY Test IVY Test was used as an additional indirect measure of program outcomes. However, there are some issues with the IVY Test report that haven t been resolved yet. The results will be reported once accurate information becomes available. Results: Key Performance Indicators The MBA program is a mature program with over 15 years of operational and sustainable history. Due to the shrinkage of the adult student market and increased competition, continued growth in size and registrations is unlikely and unexpected. Nevertheless, the program continues to be the largest graduate program and maintains sufficient registrations to remain more than viable and productive. Table 4 shows the key performance indicators of the MBA program for the recent years. 9
The number of new students in 2015-2016 increased slightly from last year. The total number of registered students decreased slightly. The total number of graduates remained almost the same as last year. The 1 st year persistence rate and graduate rate increased. The 2015-2016 retention rate remained at about the same level of previous years. The employment and related employment rates of those who graduated in the 2014-2015 academic year continued to be relatively high. Employment data of the 2015-2016 academic year are not finalized and will be updated as this information becomes available. These KPIs will be closely monitored. Table 4. MBA Program Key Performance Indicators MBA Program 2009-2010- 2011-2012- 2013-2015- 2014-15 10 11 12 13 14 16 Total New Students 342 330 314 263 216 182 219 Total Registered Students 1,156 1,085 980 887 744 596 560 Retention Rate 78.8% 79.6% 79.2% 79.8% 79.8% 80.5%* 79.7% 1st Year Persistence Rate 67.6% 72.0% 68.7% 63.8% 65.7% 56.4% 65.9% Total Graduates 307 284 256 223 225 163 165 Graduation Rate 67.1% 67.3% 67.0% 73.4% 59.8% 59.4%** 67.5% Employment Rate 91.0% 83.9% 93.2% 93.3% 99.5% 99.5% N/A Related Employment Rate 95.3% 83.1% 89.7% 87.0% 93.7% 83.2% N/A * Because the 2014-2015 KPI report was finalized after the 2014-2015 assessment report was disclosed, the 2014-2015 retention rate is updated in Table 4 of this report. ** This rate may not be accurate as the graduation rates of the BI and IS majors are missing. Progress Report on 2015-2016 Continuous Improvement Action Plan Progress on the action items in the 2015-2016 Continuous Improvement Action Plan is reported below. Continue to Improve MBA Student Learning Efficiency in Finance In Progress The leading faculty of finance revised all finance courses and economics courses. To do so, thorough research on textbooks and software tools were conducted. Adjunct faculty s input were solicited. Multimedia resources were added. Significant changes were made. In addition, a number of online labs (e.g., Connect, MyFinanceLab, etc.) have been incorporated into the semester curricula of finance courses. The next step is to measure the effectiveness of the changes. Continue to Improve Instructional Quality in the MBA Program Completed In the 2014-2015 academic year, all faculty have completed their initial Faculty Growth and Evaluation processes. Most faculty s evaluations were at least satisfactory. Unsatisfactory areas have been addressed using the Professional Growth Plans. In the 2015-2016 academic year, faculty progresses on their Professional Growth Plans were checked on an ad-hoc base. The 2015-2016 EBI survey shows that our student satisfaction on the quality of instruction is 5.90 on 10
a 1-7 Likert scale on average. It s significantly higher than the means of the Selected 6 Peers and all institutions that participate in this survey. At the end of Summer 2016, the Faculty Growth and Evaluation Process has been transferred to the Instructional Effectiveness Department. Professional development (PD) opportunities were provided to faculty, including the Baker College Annual Faculty Conference, PD webinars, PD newsletters, etc. To Increase Opportunities for Students to Connect to Practice in the MBA Curriculum Completed CGS 501 The Graduate Seminar is the first course that MBA students take in their programs. It was significantly revised. Among the many changes, a career services forum has been developed. On the forum, students will learn about various career services provided by Baker College, an information system that manages college recruiting (i.e., HandShake), how to conduct informational interviews, benefits of using LinkedIn, and their relevant professional associations. In addition, the professional development essay in this course prompts students to conduct research on those professional associations and discuss networking opportunities. All these help students to start building their career paths at the beginning of their programs rather than in the end. To Prepare for the Quarters to Semesters Conversion In Progress In order to better serve students, Baker College has decided to convert from quarters to semesters in Fall 2017. The Dean and faculty conducted research of other MBA programs, revisited program and course outcomes, revised step plans, developed course rotation schedules, developed transition recommendations for students and many other documents. Much time has been spent to convert curricula from quarters to semesters. It is ensured that all program outcomes will be covered and measured in the semester curricula. The conversion work is on schedule. To Meet New Outcomes Assessment Requirements of IACBE Completed The MBA Program s accreditation body, IACBE, issued new requirements after our 2014 outcome assessment plan was submitted. Therefore, a new outcome assessment plan was developed and submitted to IACBE in November, 2015. In early 2015, IACBE required each institution to provide direct and indirect assessment of major outcomes. We developed outcomes of each MBA major and identified direct and indirect measures of these outcomes in the 2015-2016 academic year. However, in December 2015, IACBE removed this requirement. 2016-2017 Continuous Improvement Action Plan Improvement Strategy: 11
While the assessment data indicate students are well served in the MBA program, we strive for continuous improvement. The following strategies will be implemented in the 2016-2017 academic year for this purpose. To Ensure Quality Transition from Quarters to Semesters Baker College will transition to semesters in Fall, 2017. We will ensure that all program outcome measures are transformed and that direct measures data are collected to reveal student achievements accurately and meaningfully. Continue to Improve MBA Student Learning Efficiency in Finance The leading faculty of finance will measure the effectiveness of the new finance curricula. To Refine the PO#3 Rubric The MBA Program Outcome PO#3 (Collect, interpret and analyze existing and/or original research, using quantitative and statistical tools, and use in the decision making process) is measured by the final exam of BUS678. Its rubric has four criterial levels, which is not consistent with all the other direct measure rubrics. This rubric will be reviewed and revised. To Solve the Potential IVY Test Issues The MBA Dean will communicate with the IVY test administrator to investigate whether there are any issues regarding the test and its result analysis methods. When the investigation results become available, additional initiatives may be developed. To Evaluate the Impacts of New Marketing Firm and Marketing Strategies Baker College has transitioned to a new marketing firm. A greater marketing emphasis will be placed on graduate programs. New student enrollments in the KPI table will be monitored to evaluate the impact of the new marketing firm and marketing strategies. 12
Appendix A. Center for Graduates Studies Institutional Outcomes Graduates of Baker College s Center for Graduates Studies will possess: 1. The skills necessary to contribute to their profession through active participation in scholarly and/or professional activities. 2. The broad-based discipline specific knowledge necessary to manage and advance their professional career. 3. Data driven decision making skills necessary to produce successful outcomes. 4. An awareness and appreciation for diversity and its impact on their professional environment. 5. The knowledge and ability to follow and support the ethical standards of their profession. 6. The communication skills to clearly and proficiently share ideas, knowledge and data with a wide variety of audiences. 13
Appendix B. MBA Program Outcomes Graduates of the MBA program will be able to: 1. Communicate using the advanced oral and written communication skills necessary for success in the business environment. 2. Apply standards of ethical and legal behaviors in a professional environment. 3. Collect, interpret and analyze existing and/or original research, using quantitative and statistical tools, and use in the decision making process. 4. Analyze the interrelatedness of market, economic, social and political trends, and their impact on a global environment. 5. Analyze the strategic planning process, and develop and assess strategic plans. 6. Reason analytically and apply theory across interdisciplinary boundaries to solve problems and create innovative solutions. 7. Analyze financial reports, risk management strategies and their impact on the decision making process. 8. Evaluate various leadership strategies and the implications of their use. 9. Discuss problems from diverse perspectives and analyze the impact of individual and cultural differences on the business environment. 10. Illustrate commitment to personal and professional development, community service and lifelong learning. 11. Analyze the impact of information systems and technology on a business and demonstrate the ability to make effective information management decisions. 12. Analyze the strategic impact of human resource development and management on a business. 14