Int. J. Innovation and Learning, Vol. X, No. Y, xxxx 1

Int. J. Innovation and Learning, Vol. X, No. Y, xxxx 1 Improving MIS education in an online learning environment through course-embedded measurement June Lu*, Linda A. Hayes and Chun-Sheng Yu School of Business Administration University of Houston Victoria 14000 University Blvd. Sugar Land, TX 77479, USA E-mail: luj@uhv.edu E-mail: hayesl@uhv.edu E-mail: yuc@uhv.edu *Corresponding author Abstract: Enrolment decline in Information Systems (IS)-related programmes has been a hot issue in academia and this affects the prominence of the Management Information Systems (MIS) course in the Masters of Business Administration (MBA) core curriculum. This paper uses a course-embedded measurement model based on Association for Accrediting Collegiate Schools of Business (AACSB) programme assessment steps to investigate the possibility of attracting more online students to MIS courses via providing quality education in the Online Learning Environment (OLE). The data analysis results reveal that course-embedded measurement and the related closing-loop efforts assure learning in MIS classes in an OLE mode. Providing quality OLE classes seems to be an effective channel for enhancing and maintaining MIS enrolment and, in turn, contributes to the double-loop learning and continuous improvement desired in the entire MBA programme. Keywords: management information systems; MIS; course-embedded measurement; programme assessment; online learning; double-loop learning; innovation; learning. Reference to this paper should be made as follows: Lu, J., Hayes, L.A. and Yu, C-S. (xxxx) Improving MIS education in an online learning environment through course-embedded measurement, Int. J. Innovation and Learning, Vol. X, No. Y, pp.000 000. Biographical notes: June Lu is an Associate Professor of Management at the School of Business Administration of the University of Houston Victoria, USA. She received her Doctorate from the University of Georgia. Her specialties are management information systems and e-commerce. Her research spans wireless mobile technology acceptance and electronic/mobile commerce in different cultural settings and the effectiveness of using online learning tools in MIS education. She has published a number of articles in the Journal of Strategic Information Systems, Information & Management, the Journal of Internet Research, the Journal of Computer Information Systems, the International Journal of Mobile Communications and other journals. Copyright 200x Inderscience Enterprises Ltd.

2 J. Lu, L.A. Hayes and C-S. Yu Linda A. Hayes is an Associate Professor and the Director of Programme Assessment and Online Services of the School of Business Administration of the University of Houston Victoria, USA. She received a BSME from Clarkson University, an MBA from the University of Houston and a PhD from the University of California at Berkeley. She has 15 years of industry experience. Her research interests include decision making, judgement and perceptions. She was a NASA Summer Faculty Fellow. Recently, she has published in the Journal of Business Ethics, the Journal of Management Development, the Journal of International Marketing and the International Journal of Mobile Communications. Chun-Sheng Yu is an Associate Professor of Management at the University of Houston Victoria, USA. He holds a DBA from Mississippi State University. His research interests comprise cross-cultural management, quality management and mobile commerce. His articles have appeared in a variety of management journals, including the International Journal of Organizational Analysis, the Journal of Strategic Information Systems, Information & Management, Quality Management Journal, Current Topics in Management, the Journal of Internet Research, the Journal of Computer Information Systems, the International Journal of Mobile Communications and other journals. 1 Introduction Enrolment decline in Information Technology (IT) and Information Systems (IS)-related academic programmes has been a hot issue in academia. While it is reasonable to believe that the breaking of the dot-com bubble, the significant reduction in job opportunities as a consequence of a declining economy, and the offshoring of IT functions may not directly affect the role of the Management Information Systems (MIS) course in the Masters of Business Administration (MBA) core, they more or less affect the prominence of this course in the curriculum (Rob, 2003). At the same time, online MBA programmes are mushrooming with increasing student enrolment thanks, in part, to improvements in the virtual learning environment and the passion for life-long learning. The current trend of providing online or web-based MBA programmes by a growing number of universities has placed delivery of the MIS course in an OLE mode (Lu et al., 2003). Thus, the online MIS course in an MBA programme is facing the challenge of providing quality learning to meet the expectations of the public and to maintain the increased level of enrolment. MBA programmes offered by an Association for Accrediting Collegiate Schools of Business (AACSB) accredited business school are under regular programme assessment, which has a dual focus of accountability and continuous improvement (Davis et al., 2007). In the AACSB International (2008a) standards, assessment refers to quantifying student learning by measuring learning outcomes to provide evidence of student achievement and to guide future instruction (Harich et al., 2005). Such assessment is based on periodic measurement of student performance for each programme s learning goals. Consequently, AACSB programme assessment is used to determine the success or failure of an entire programme by forming a feedback loop of measuring the impact of instruction on student performance and suggesting programme improvements (Shaftel

Improving MIS education in an online learning environment 3 and Shaftel, 2007). The MIS course in the core curriculum must coordinate with other MBA courses in complying with AACSB standards and in providing any programme learning goals that are covered by MIS content and materials, since AACSB standards focused on programs not individual courses. Such regular programme assessment provides an excellent opportunity for MIS faculty to measure student learning and make improvements on a continuous basis. A preliminary literature review reveals little information on how AACSB programme assessment data is helping to measure the appropriateness of course focus, quality of student learning, and areas of continuous improvement for the MBA core MIS course. The focus of this paper is to explain how course-embedded measurements, as part of the AACSB programme assessment, can foster continuous improvement of the online MIS course in an MBA programme. Specifically, the paper will answer three questions: 1 Can OLE-based MIS learning outcomes meet the AACSB assessment expectations? 2 Do interventions based on previous assessment results make a significant difference in online MIS student learning performance? 3 Does the pattern of performance enhancement continue over time? To answer these questions, this paper reviews the relevant literature and proposes a research model guiding course enhancements and continuous improvement. The remainder of the paper describes the course assessment and enhancement process in a university in Coastal Bend region of Texas, provides data analysis results, and offers a discussion of contributions and recommendations for future efforts. 2 Literature and research model The terms internet-based learning, web-based learning have been used interchangeably in academia to refer to learning in the OLE mode that has typically been asynchronous, learner centred, web-based, and multimedia enhanced. Learning in the OLE mode has three dimensions of: technology, access and quality (Edwards and Finger, 2007). The major characteristic of the technology dimension is the integration of multimedia and hyperlinks through the web to produce hypermedia (Rossiter and Crock, 2006) which enables interactions between learners and content, between learners and instructors, and between learners and learners in the learning process by eliminating the limitations of time and space (Katz, 2000; Katz, 2002; Moore and Kearsley, 1996). The access dimension refers to the choices a learner can have: she can choose between online or offline, synchronous or asynchronous learning mode. The current trend is moving towards blended learning which allows students to combine location-independent, electronically-mediated approaches with place-specific, real time teaching and learning experiences, with the objective of optimising the learning outcome and cost of programme delivery (Mahadevan et al., 2002; Singh and Reed, 2001). The quality dimension is best encapsulated in two complementary learning approaches which share a common focus on the learner, lifelong learning and learner-centred learning.

4 J. Lu, L.A. Hayes and C-S. Yu Due to technological advances, growing college populations, and changing student profiles, higher education institutions have seen remarkable growth in learning in the OLE mode over the decade, and such growth is expected to continue. Business education is the most popular area for online education. 2.1 Growth of online MBA programmes Despite the fact that enrolment at many traditional business schools has dropped, online MBA courses and programmes have become one of the fastest growing segments in higher education. In year 2006, 47 of the 61 schools offering online MBA degrees profiled on the BusinessWeek website were AACSB-accredited (Zammuto, 2008). By 2007, according to GetEducated.com, a web directory providing online MBA rankings, 168 online MBA programmes were offered through accredited universities in the USA. Simply stated, educational delivery using the internet has become part of the way that business schools do business. Online MBA programmes usually use one of the two delivery modes. One is the pure online mode whereby all teaching, content, and interactions occur virtually. Such programmes use a variety of instructional technologies, including web-based courseware such as WebCT and Blackboard (currently both the products of Blackboard, Inc.), asynchronous discussion boards, web conferencing, video conferencing, e-mail, instant messaging, and audio bridges. In contrast, the second or blended model utilises asynchronous online and instructional technologies combined with face-to-face (f2f) synchronous components (Hochberg, 2006). Students from both online and offline programmes are being awarded the same degree. Are they receiving the same quality of education? This question has been on the minds of numerous business employers and employees interested in online instruction. This has also been a hot topic among academic researchers. A recent review of 355 research studies and reports has revealed some optimistic findings that when campus learning is compared to virtual learning there are no significant differences in learner outcome or satisfaction (Phillips, unknown). However, research studies also pointed out that each delivery method enhances different, but equally valuable, academic skill sets, and that many factors affect the overall quality of an educational experience (e.g., Arbaugh, 2007; Lu et al., 2003; Phillips, unknown). Therefore, ensuring a quality online learning experience has been an important task for accredited schools. 2.2 MIS in MBA programmes MIS is generally ascribed to be the field of managing behavioural issues as well as technical issues surrounding the development, use, and impact of information systems in organisations (Laudon and Laudon, 2006). MIS in this paper refers to an introductory level IS course included in the core curriculum of many MBA programmes in the USA. Graduate students in this course are usually required to establish a basic understanding of a number of IS applications in various business situations at numerous managerial levels. While technical proficiency is important, business firms are aggressively seeking professionals with project management expertise, enterprise and industry knowledge, and business skills necessary for customer-facing roles (Overby, 2006). Because of this trend, some institutions have taken curriculum-based approaches to steer their programmes to cope with these needs and to reduce declining enrolments (Shore and Briggs, 2007).

Improving MIS education in an online learning environment 5 Accordingly, the MIS course is expected to show students how various management information systems are helping to achieve organisational competitive advantages in managing business processes and operations, enabling data driven decision making, performing internet marketing, assisting supply chain management, and playing a critical role in strengthening business ethics (Laudon and Laudon, 2006). However, many students and employees tend to equate this IS course with learning software tools and programming. It often takes considerable effort to correct this misimpression. 2.3 AACSB programme assessment Assessment is a key to quality assurance and to an ongoing improvement process. The AACSB International (2008a; 2008b) standards adopt a programme evaluation approach congruent with Tyler s objectives-oriented educational evaluation process, which comprises establishing major programme goals, defining objectives to meet the goals, developing measures for the objectives, collecting data, and then comparing performance on the objectives to the original goals (Fitzpatrick et al., 2004). In the AACSB (2008c) standards, assessment is used by business schools to indicate the process of measuring learning outcomes to provide evidence of student achievement and the effectiveness of programme curriculum (Harich et al., 2005). Such a student performance-based assessment process forms a feedback loop with programme improvement efforts. The ultimate conclusion is concerned with providing information to assist in making judgements about programme accreditation, continuation, or suspension. Assessment, as used in educational settings, is a component of the broader concept of evaluation. There are three basic approaches to assessment: (1) reputation, (2) resources, and (3) outcomes (Astin, 1987). Outcomes-based assessment focuses on the results of the education process by measuring student academic achievement at the end of a learning unit. It provides a way of evaluating how well educational programmes are doing and identifies ways to further enhance them. To make assessment judgements in a systematic way, information is collected, analysed, and interpreted in order to make inferences about student learning performance and development (Herring and Izard, 1992; JCSEE, 2003). This method provides the metrics to determine if there has been any improvement in student performance and is credible in establishing immediate feedback on the quality of an education programme. AACSB standards call for such Outcomes-based Assessment (OA) as a result of the accreditation requirements of the US Department of Education (Apostolou, 1999). AACSB International (2008c) assessment practices include direct measures of achievement based on examples of student performance on tests or course project work. Many college and university business schools provide direct outcome evidence of students progress in meeting learning goals and modify their degree programmes in light of their findings (Trapnell, 2005). Such OA can be broadly defined as a procedure in which students complete tasks or processes that demonstrate their ability to apply knowledge and skills, or put knowledge and understanding into action in simulated or real-life situations (Muraki et al., 2000). One form of OA often used in AACSB accredited schools is course-embedded measurement such as writing samples, oral presentations, or project activities. Such course-embedded measurements often are already required within course context and can be used to meet programme evaluation goals (Shaftel and Shaftel, 2007).

6 J. Lu, L.A. Hayes and C-S. Yu An assessment-based approach to MBA instruction throughout the curriculum can clarify the placement of courses among other mutually reinforcing courses for developing overall student business application capability. This objectives-oriented, course embedded, performance-based assessment process provides an excellent opportunity to gather meaningful information for adjusting direction, content delivery and pedagogy used in MIS, a core MBA course. To ensure the quality and continuity of such a performance-based assessment process, a theory framework is proposed for further understanding and guidance. 2.4 Relevant theory model Educational assessment is the process of determining whether students are actually learning what they are expected to learn in a programme and enhancing the programme to improve student performance (Martell, 2005). Based on this belief, AACSB has defined a five-step assessment process model (AACSB International, 2008c): Step 1 Step 2 Step 3 Step 4 Step 5 defining programme learning goals and objectives aligning curriculum with goals identifying instruments and measures collecting, analysing and disseminating assessment data using assessment data for continuous improvement. While this model indicates the general stages for a course-embedded measurement to go through in the entire assessment process, this model does not fully explain how to achieve desired double-loop learning nor does the model mention any specific suggestions about improvement. Perhaps because of a common belief of model completeness at the point of data analysis and presentation of results for assessment purposes, many traditional assessment models do not provide adequate direction for programme improvement. In contrast, the transformative assessment approach is designed to address the need to close the loop (Lorenzetti, 2004). Transformative assessment promotes use of rubrics to track learning activities for identification of where the programme is truly transformative and where it may fall short. Clearly this type of assessment has the potential to provide continual feedback for incremental adjustments and should occur continuously throughout the life of a programme. Transformative assessment is thus held as a new and richer form of assessment. Some researchers believe that this assessment method is particularly important to distance or online education where the risks of failure tend to be high. Garrison et al. s (2000) Community of Inquiry (CoI) model of online learning contends that effective online learning is formed by three types of presence: social, teaching, and cognitive presence. While social presence sets the climate for learning in an online community and cognitive presence represents the inquiry process of a learner, it is the teaching presence that does the job of blending learning. This model conceptualises three components of teaching presence toward expected double-loop learning in OLE: 1 course design and organisation 2 facilitating discourse 3 direct instruction (Anderson et al., 2001).

Improving MIS education in an online learning environment 7 This newly-emerged CoI model has been tested, within limited scope, among MBA students and won empirical data support in the context of management education (Arbaugh, 2007; Arbaugh and Hwang, 2005; Arbaugh and Hwang, 2006). It is believed to be a framework of great potential for studying online management education and rather robust for accommodating future technological, pedagogical, and organisational changes (Garrison and Anderson, 2003). According to this framework, design and organisational aspects of teaching presence relate to the planning and design of the structure, process, interaction and evaluation aspects of the online course. Teaching activities are concerned with creating a clear and consistent course structure to provide the context for meaningful discourse and direct instruction. Facilitating discourse is the means by which students are engaged in interacting about and building upon the information provided in the course instructional materials. Facilitating discourse requires the instructor to review and comment upon student inputs, raise questions and make observations to move discussions in a desired direction. Unlike design and organisation, facilitating discourse must be accomplished by the instructor in concert with a community of learners (Anderson et al., 2001; Rovai, 2001). While an effectively designed course creates opportunities for an online community, the community will not be created unless participants actually engage each other. This aspect of teaching presence is, therefore, critical to attain the learning objectives in the course particularly at the graduate-level of education. Anderson et al. (2001) contextualised direct instruction as the component of teaching presence where instructors provide intellectual and scholarly leadership in part through the sharing of their subject matter knowledge with the students. Direct instruction covers diagnosing comments and misperceptions for accurate understanding, injecting sources of information, directing discussions in useful directions, and scaffolding learner knowledge to raise it to a new level. These components are separate and discernable components (Arbaugh and Hwang, 2005). Yet, they work together to enable double-loop learning a distinctive characteristic of graduate-level online management education (Rungtusanatham et al., 2004). 2.5 Operational research model Direct measurement efforts can provide substantive evidence for making programme improvements. It is up to the instructional intervention to achieve the double-loop learning as desired in management education. Instructional intervention is an important modifying factor over learning content and learning environment to strengthen or weaken the expected learning outcomes and satisfaction (Martinson and Cole, 2002). To answer the research questions, we needed a framework to serve as a blue print for the OLE-based MIS course measurement and improvement effort. Figure 1 displays this operational model framework. Based on a review of the relevant theories, we began by drawing on the five-step AACSB process model, which provides a general structure for developing and using course-embedded measurement for programme assessment. For identification of problem areas, the idea of rubrics in transformative assessment was employed in our model step 3. For step 5 Using Assessment Data for Continuous Improvement by generating ideas for programme and course enhancements we adopted the three dimensions of teaching presence from the CoI model to create methods for improving online teaching in MIS. To make sure that the assessment is used as a means

8 J. Lu, L.A. Hayes and C-S. Yu to an end, not as an end in itself, we added step 6 Execute Instructional Intervention. We added a back arrow to step 1 for programme enhancements and a back arrow to step 3 for course enhancements. Thus, the model provides two loops or a double double-loop for guiding continuous improvement efforts. We also provided a back arrow to step 4 for recollecting and reanalysing student performance in specific course embedded measures. This third loop provides a step for course measurement reliability and consistency. Figure 1 Course-embedded measure and improvement model Step 1 Define Program Learning Goals Step 2 Align Curriculum with Learning Goals Step 3 Identify instruments and course embedded measures 3a Develop/Modify MIS rubric 3b Align rubric to program learning goals 3c Develop student assignment 3d Specify expectations Step 4 Collect MIS student data and analyze results Step 5 Generate ideas for continuous improvement 5a Course design and organization 5b Facilitating discourse 5c Direct instruction Step 6 Execute instructional intervention and measure results 6a Course design and organization 6b Facilitating discourse 6c Direct instruction

Improving MIS education in an online learning environment 9 With the operational model shown in Figure 1, we tested the following three hypotheses to provide data for answering our research questions: H 1 Student performance in online MIS classes meets or exceeds learning expectations on the four rubric traits. Ho 1 There are no statistically significant differences in online MIS classes between actual student performance scores on the four rubric traits and learning expectations. H 2 The mean measurement scores on the four rubric traits in the online MIS classes after instructional intervention were statistically significantly higher than those of their counterparts before instructional intervention. Ho 2 There are no statistically significant differences between the mean measurement scores in online MIS classes on the four rubric traits across three consecutive semesters. 3 Course-embedded measurement process 3.1 Preparation and measurement Under the external pressure of meeting the AASCB accreditation standards and driven by two internal drives of: 1 integrating MIS more effectively with the other MBA core courses 2 ascertaining the effectiveness of the MIS course in OLE, we followed the operational model in the MIS course-embedded measurement process at a regional university in southwest of Texas since Fall 2006. MIS in an MBA core is designed to help students understand how MIS provides businesses and organisations with competitive advantages in managing operations, in decision making, in strengthening marketing and supply chain management (Shore and Briggs, 2007). With this focus, a rubric was created for MBA students to apply a firm-based value chain model with decision making in analysing IS/IT implementation in the functional areas and processes of an organisation. This rubric was compared and adjusted to the relevant MBA programme learning goal that MBA students should be able to analyse business operations and processes. Based on this rubric, four traits defining learning expectations were developed as shown in Table 1. The MIS students were given an assignment to prepare a paper to address the four traits. For step 4, we collected data from the MIS online sections at the end of the Fall 2006 semester. In early Spring 2007, a team comprised of faculty teaching the MIS course and the school s Director of Program Assessment met to assess student performance. To assure fairness, student names were removed from all papers for assessment by a staff member beforehand. Since this was the first time the MIS course-embedded measurement was used for programme assessment purpose, an overall goal was set that 75% of students meet or exceed expectations for each of the four traits.

10 J. Lu, L.A. Hayes and C-S. Yu Table 1 Rubric and traits for MIS course Trait Does not meet expectations (1) Meets expectations (2) Exceeds expectations (3) Describe firm-based value chain model and decision-making levels. Failure to identify the most important features. Description of most important features. Complete and correct description of the value chain model and decision-making levels. Apply the value-chain model and decision-making level identification to the specific firm s situation. Failure to describe the value chain activities and decision-making levels in a firm. Brief description of the value chain activities and decision-making levels in a firm. Complete and accurate description of the value chain activities and decision-making levels in a specific firm. Analyse the opportunities in terms of functional areas, business process(es), and decision levels for IS/IT implementation in the firm. Failure to analyse the functional areas, business process(es), and decision levels in need of IS/IT implementation. Brief analysis of the functional areas, business process(es), and decision levels in need of IS/IT implementation. Comprehensive and logical analysis of the functional areas, business process(es), and decision levels in need of IS/IT implementation. Analyse the matching functionality of the IS/IT product(s). Failure to analyse the matching functionality of the IS/IT product(s). Briefly analysis of the matching functionality of the IS/IT product(s). Comprehensive and logical analysis of the matching functionality of the IS/IT product(s). Notes: Apply firm-based value chain model and level of decision making in analysing the opportunities for IS/IT implementation in an organisation, on the basis of proper understanding of the relationship between information system functionality and the functional areas and processes in that organisation. Result

Improving MIS education in an online learning environment 11 One evaluator scored the student paper on each trait first. The raw scores were recorded using a 3-point scale with 1 for under expectation, 2 for meeting expectations, and 3 for exceeding expectation. The second evaluator reviewed all the scored papers. Certain adjustments were made to ensure minimum evaluator bias. All the scores on each trait were then tallied to obtain the needed percentages. 3.2 Data analysis results To verify whether students in online MIS classes meet or exceed learning expectations, we first calculated the percentages of those student papers that did not meet expectations and those that met or exceeded expectations on each of the four rubric traits for Fall 2006. The results of the assessment are shown in Table 2. In summary, online students did meet the overall goal of 75% of students meeting or exceeding expectations for traits 1 and 2, but fell short on traits 3 and 4. Online students could describe and basically apply concepts but did not do as well when analysing the IS/IT implementation. Hypothesis 1 is thus partially rejected. Table 2 Online student learning performance frequency (percentage) in Fall 2006 (N = 30) Trait Bloom s learning level Student performance below expectations Student performance meets or exceeds expectations Expectations of student performance (%) 1. Describe firm-based value chain model and decision-making levels. 2 4 (13.3%) 26 (86.7%) 75 2. Apply value-chain model and decision-making level identification to the specific firm s situation. 3 7 (23.3%) 23 (76.7%) 75 3. Analyse the opportunities in terms of functional areas, business process(es), and decision levels for IS/IT implementation. 4 10 (33.3%) 20 (66.7%) 75 4. Analyse the matching functionality of the IS/IT product(s). 4 17 (56.7%) 13 (43.3%) 75 To examine the first null hypothesis whether actual learning performance scores on the rubrics from the online MIS class are significantly different from learning expectations, a chi-square test shows no statistically significant difference between the actual learning performance of students for traits 1, 2 and 3 and learning expectations. However, the chi-square test for trait 4 reveals a statistically significant difference between expected and actual student performance (p < 0.01). The null hypothesis, Ho 1, was thus partially rejected.

12 J. Lu, L.A. Hayes and C-S. Yu 4 Closing the loop 4.1 Designing and executing instructional interventions With the course measurement results, we examined the online MIS course using the three components of Teaching Presence promoted in CoI model (Garrison et al., 2000). We first revisited the course design and organisation. Weaknesses were identified with some class assignment descriptions. Even though the trait requirements were present in the assignment document, the specifics such as which part of the assignment should be devoted to demonstrate what trait and the expected length of the assignment were not provided. Besides revising the assignment description documents, we also posted an exemplary student paper from the previous semester with some hints for students to see how to demonstrate their capability on the four traits of the rubric. In addition, to help online students to fully grasp what the assignment entailed, especially for higher Bloom s (1956) level tasks, we decided to use a case study on the WebCT discussion board for open practice on achievement of the four traits. Through the facilitating discourse of discussion and feedback, the instructor could realise and adjust via feedback the progress the online students were making in reaching the expectations. Considering that some students might learn better through real-time social interaction because of personality and character trait differences (Lu et al., 2003), Macromedia Breeze Meeting (presently Acrobat Connect) was selected to provide for direct instructions and communications in synchronous mode. Macromedia Breeze Meeting is a web conferencing application programme. This programme allows one to present to, meet with, or collaborate with students, colleagues and customers over the internet in real time on numerous platforms. One can create an online meeting room by assigning a unique URL. Participants attend a meeting by using their browsers to visit the meeting room URL. This programme provides broad customisation options as well. After creating a virtual meeting room in Breeze, one can specify how material appears in the attendees browsers, determine the kind of content to use, bring PowerPoint slides, digital video, FlashPaper documents, Flash simulations, and other types of media into your live meetings. One can interact and collaborate using live video and voice broadcasts, chat messages, whiteboard drawings and annotations, and demonstrations of applications on your computer. 1 Business organisations found this web meeting programme ideal for project reviews, employee training, prospective employee interviews, contract negotiations, and keeping in touch with offshore resources (Heck, 2005). To perform the instructional interventions in the online MIS classes, online students in Spring 2007 were given the enhanced assignment instructions to better explain what students were to do. Students were informed what type of performances met the assignment expectations and what type did not. Vonderwell et al. (2007) found that online students value online discussions. An example textbook case with questions directly concerning the four traits under measurement was discussed in WebCT. Instructor feedback on how well students handled each portion was provided in detail. To strengthen the learning effectiveness, we modified our pure asynchronous MIS online teaching in WebCT by incorporating Macromedia Breeze Meeting Add-in 2005 version to open real-time audio, video enabled virtual meeting sessions.

Improving MIS education in an online learning environment 13 The Breeze Meeting programme allowed the instructor and the online students to enter a virtual meeting room where they used their headsets and computer cameras to communicate with each other and with the instructor real time. The virtual meeting participants also used the provided chat room to do text-based conversations among themselves. In this virtual meeting room the instructor was able to share her own computer desktop and important documents with the students on her own computer desktop. Thus, everyone at the meeting session could see clearly and real-time the focus of certain assignments and demonstrations of the hands-on skills. We scheduled seven Breeze meeting sessions with the online MBA students taking the WebCT MIS classes during Spring 2007, roughly once every two weeks. Those meeting sessions mostly took place on weekends to accommodate students who were also full-time job holders. Student feedback on adding this virtual meeting tool was very positive. Students found the addition of such a real-time communication tool extremely helpful in deepening their understanding of course material as well as enhancing their ability to learn in an online environment. 4.2 Data analysis results We collected student papers from online MIS classes in Spring 2007, assessed them, and compared the learning outcomes on the predetermined assessment traits with the results from Fall 2006. The descriptive results of this assessment are in Table 3. Online students improved their performance on every assessment trait and over 75% of online students met or exceeded learning expectations on all four traits. Four T tests were then performed on the student trait raw scores between the online section in Fall 2006 and the counterpart in Spring 2007. The T tests show that the mean scores of the online MBA students in Spring 2007 were statistically significantly higher than those from Fall 2006 on all the four traits (p 1 < 0.01, p 2 < 0.05, p 3 <.05, p 4 < 0.000). The second directional hypothesis, H 2, is thus accepted. This result supports the notion that the instructional intervention covering the three components of Teaching Presence help to bring about significant improvement in online student performance. The same instructional interventions were kept in the online MIS sessions for Fall 2007 and Spring 2008. To examine the reliability and consistency of the online MIS student learning outcomes across time, the same course-embedded assessment measurement rubrics and the same course assignment description were used, and the assessment was conducted by the same evaluators using the same procedures as in Spring 2007. The relevant descriptive data are contained in Table 3. The resulting scores were then compared with those in Spring 2007 by using One-way ANOVA procedures to test the last hypothesis. Despite the fluctuation in the raw scores across the semesters, the F tests did not reveal statistically significant differences on any of the four traits (p 1 > 0.53, p 2 > 0.24, p 3 > 0.26, p 4 > 0.21) across the time. These results indicate that the same instructional intervention that was used through three consecutive semesters obtained similar student performance results. The last hypothesis, Ho 2, therefore, is accepted.

14 J. Lu, L.A. Hayes and C-S. Yu Table 3 Online student learning performance frequency (percentage) a comparison between four consecutive semesters Trait 1. Describe firm-based value chain model and decision-making levels. 2. Apply value-chain model and decision-making level identification to the specific firm s situation. 3. Analyse the opportunities in terms of functional areas, business process(es), and decision levels for IS/IT implementation. 4. Analyse the matching functionality of the IS/IT product(s). Fall 2006 N = 30 Below expectation Meet/Exceed expectation 4 (13.3%) 26 (86.7%) 7 (23.3%) 23 (76.7%) 10 (33.3%) 20 (66.7%) 17 (56.7%) 13 (43.3%) Spring 2007 N = 40 Below expectation Meet/Exceed expectation 2 (5%) 38 (95%) 3 (7.5%) 37 (92.5%) 7 (17.5%) 33 (82.5%) 3 (7.5%) 37 (92.5%) Fall 2007 N = 46 Below expectation Meet/Exceed expectation 1 (2%) 45 (98%) 1 (2%) 45 (98%) 4 (8.7%) 42 (91.3%) 5 (10.9%) 41 (89.1%) Spring 2008 N = 29 Below expectation Meet/Exceed expectation 3 (11.5%) 26 (89.5%) 4 (13.8%) 25 (86.2%) 6 (21%) 23 (79%) 5 (17.2%) 24 (82.8%)

Improving MIS education in an online learning environment 15 5 Limitations and contributions While our study results may have important implications for continuous improvement in online MIS instruction, it is important to acknowledge the following limitations prior to discussing the contributions: The findings from this study were from the MBA level MIS course in Texas. Considering the limited sample size and the student profiles (mostly from the Coastal Bend region of Texas), we caution generalising these findings to other MIS courses in other MBA programmes. The course embedded assignment was a project where students researched highly varied organisational types and business natures. As a result, this type of student project may add to the difficulty of students achieving learning expectations on those traits that reflect higher Bloom s learning levels. This might have affected our assessment results. Although we recognise that MIS courses in different MBA programmes and on different university campuses may be taught differently, we believe this assessment and continuous improvement model is replicable, and can be adapted by other MIS courses. In fact, it should be a direction for further research in this regard. One feasible approach is to repeat the cycle as illustrated in the model at regular intervals. Then, reexamine course design and organisation, facilitating discourse, and direct instruction after each measurement. The most important, of course, is to execute instructional interventions in any of these three areas in real online delivery. The contributions made by this study are primarily in three aspects: First, we enhanced the AACSB assessment model by adding the double-loop learning concept. This enhanced model specifies the tasks at each operational step for conducting assessments and for making improvements on a continuous basis. This model successfully guided the process in our study to achieve double-loop learning as desired in MBA education. Second, with a focus on assuring quality learning in online sections of the MBA level MIS class, this study has revealed the value of course-embedded measurement as a part of the MBA programme assessment from two perspectives: accountability and continuous improvement. The goal of assessment in online MIS classes is nearly identical to traditional classes. Both formats address student learning by providing reasonably valid and reliable information that is feasible to collect, given available knowledge and resources. With such information in hand, online sessions of an MIS class can improve students learning and report effectively to other audiences the outcomes. This study seems to support Arbaugh and Hwang s (2005) argument that in a highly demanding OLE, instructor online experience appears to be a stronger predictor than disciplinary effects. For lack of real-time, face-to-face communications in the OLE mode, instructions need to be reviewed carefully and continuously to ensure that teaching presence is adequate in terms of structure and organisation, facilitating discourse, and direct instruction. Online students thus are able to understand what the learning tasks entail. For MIS classes delivered entirely in an OLE mode whereby all teaching, content, and interactions occur virtually, great care must be taken to make sure that instructions are conducted in a way to facilitate student comprehension of the learning objectives and specific expectations.

16 J. Lu, L.A. Hayes and C-S. Yu Third, this study provided some strong support for integrating synchronous web meeting sessions into MIS online teaching. Numerous researches have found learning most effective in a combined mode of synchronous and asynchronous communications. Web meeting wares are playing an increasingly important role in IS project management and training in the business world (Butcher, 2005; Heck, 2005; Hochberg, 2006). Instructors thus should be encouraged and trained to use a variety of instructional technologies effectively, including web-based courseware, web conferencing, and other multimedia tools. Research has revealed that an asynchronous virtual learning environment does not cope well with social and procedural learning as desired by many online students (Eastman et al., 2003). Proper integration of various virtual learning tools helps to enrich virtual learning with multiple communication channels. Students of different learning styles thus can be more fairly evaluated. Students can choose to enjoy the flexibility promised by the asynchronous mode of online learning and instant communications enabled by real-time multimedia tools. On the other hand, for an MIS online class, a good combination of a variety of virtual learning tools gives students first-hand experience of using a number of IT products and help them realise how IS is changing and benefiting our life. With assessment experience gained in the online MIS classes, it is highly recommended that such assessment and remedy efforts be conducted at regular intervals. Using the same type of assignment or assessment test across different learning environments is also recommended for accurate comparisons. More than one assessment method can also be considered to further verify reliability. 6 Conclusions The experiments with the MIS classes in OLE mode in our study indicate that educational assessment efforts to satisfy AACSB standards can be used for effective double-loop learning at both the programme and course levels, and thus enhance student learning outcomes and satisfaction. The study results give us some confidence to conclude that with appropriate instructional interventions in course design and organisation, facilitating discussions, and direct instruction, MBA students can meet and exceed learning expectations in an OLE-based MIS class. Properly designed and executed instructional interventions between course-embedded measurements in the long run help with continuous improvement. References AACSB International (2008a) Accreditation standards, http://www.aacsb.edu/resource_centers/ assessment/standards.asp (retrieved 1 August 2008). AACSB International (2008b) Eligibility procedures and accreditation standards for business accreditation, http://www.aacsb.edu/accreditation/process/documents/aacsb _STANDARDS_Revised_Jan07.pdf (retrieved 1August 2008). AACSB International (2008c) The assessment process, http://www.aacsb.edu/resource_centers/ assessment/overview-process.asp (retrieved 1 August 2008). Anderson, T., Rourke, L., Garrison, D.R. and Archer, W. (2001) Assessing teaching presence in a computer conferencing context, Journal of Asynchronous Learning Networks, Vol. 5, No. 2, http://www.aln.org/publications/jaln/v5n2/v5n2_anderson.asp.

Improving MIS education in an online learning environment 17 Apostolou, B.A. (1999) Outcomes assessment, Issues in Accounting Education, Vol. 14, pp.177 197. Arbaugh, J.B. (2007) Does the community of inquiry framework predict outcomes in online MBA courses?, Academy of Management Proceedings, pp.1 6. Arbaugh, J.B. and Hwang, A. (2005) Confirmatory study of teaching presence in online MBA courses, Academy of Management Proceedings, pp.a1 A6. Arbaugh, J.B. and Hwang, A. (2006) Does teaching presence exist in online MBA courses?, Internet & Higher Education, Vol. 9, No. 1, pp.9 21. Astin, A.W. (1987) Assessment, Value-Added and Educational Excellence in Student Outcomes Assessment: What Institutions Stand to Gain, edited by D.T. Halpern, San Francisco, CA: Jossey-Bass, Inc., pp.89 107. Bloom, B. (1956) Taxonomy of Educational Objectives, Handbook I: Cognitive Domain, New York: Longman, Green and Company. Butcher, D.R. (2005) Macromedia s solution is blowin in the wind, Customer Inter@ction Solutions, Vol. 23, No. 11, pp.24 25. Davis, N.T., Kumtepe, E.G. and Aydeniz, M. (2007) Fostering continuous improvement and learning through peer assessment: part of an integral model of assessment, Educational Assessment, Vol. 12, No. 2, pp.113 135. Eastman, J.K., Swift, C.O., Bocchi, J., Jordan, R. and McCabe, J. (2003) The challenges and benefits of teaching MBA courses online: lessons from the Georgia WEBMBA, AMA Winter Educators Conference Proceedings, Vol. 14, pp.153 154. Edwards, A. and Finger, G. (2007) elearning and sport management: hyperpedagogy possibilities, Sport Management Review, September, Vol. 10, No. 2, pp.191 208. Fitzpatrick, J.L., Sanders, J.R. and Worthen, B.R. (2004) Program Evaluation: Alternative Approaches and Practical Guidelines, 3rd ed., Boston, MA: Pearson. Garrison, D.R., Anderson, T. and Archer, W. (2000) Critical inquiry in a text-based environment: computer conferencing in higher education, The Internet and Higher Education, Vol. 2, pp.87 105. Garrison, D.R. and Anderson, T.E. (2003) Learning in the 21st Century: A Framework for Research and Practice, London: RoutledgeFalmer. Harich, K.R., Fraser, L. and Norby, J. (2005) Taking the time to do it right: a comprehensive, value-added approach for assessing writing skills, in K. Martell and T. Calderon (Eds.) Assessment in the Disciplines, Vol. 1, No. 2, Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way, Tallahassee, FL: Association for Institutional Research, Florida State University. Heck, M. (2005) It s like being there, virtually, InfoWorld, 9 May, Vol. 27, No. 19, pp.24 29. Herring, H.C., III and Izard, D. (1992) Outcomes assessment of accounting majors, Issues in Accounting Education, Vol. 7, pp.1 17. Hochberg, J. (2006) Online MBA programs: emulating global business, Distance Education Report, Vol. 10, No. 2, pp.7 8. Joint Committee on Standards for Educational Evaluation (JCSEE) (2003) The Student Evaluation Standards: How to Improve Evaluations of Students, Thousand Oaks, CA: Corwin Press. Katz, Y.J. (2000) The comparative suitability of three ICT distance learning methodologies for college level instruction, Educational Media International, Vol. 37, pp.25 30. Katz, Y.J. (2002) Attitudes affecting college students preferences for distance learning, Journal of Computer Assisted Learning, Vol. 18, pp.2 9. Laudon, K.C. and Laudon, J.P. (2006) Management Information Systems: Managing the Digital Firm, 10th ed., Upper Saddle River, NJ: Pearson Prentice Hall. Lorenzetti, J. (2004) Transformative assessment in higher education, Distance Education Report, 15 March, Vol. 8, No. 6, pp.3 7.

18 J. Lu, L.A. Hayes and C-S. Yu Lu, J., Yu, C.S. and Liu, C. (2003) Learning style, learning patterns and learning performance in a WebCT-based MIS course, Information & Management, Vol. 40, No. 6, pp.497 507. Mahadevan, V., Braun, R.M. and Kadi, A. (2002) A perception of next generation elearning in Australia, in M. Driscoll and T.C. Reeves (Eds.) E-Learn 2002, Montreal: Association for the Advancement of Computing in Education, pp.1835 1838. Martell, K. (2005) Assessment in business schools: what it is, where we are going, and where we need to go now, assessment of student learning in business schools: best practices each step of the way, in K. Martell and T. Calderon (Eds.) The Association for Institutional Research and AACSB International, Vol. 1, No. 1, pp.1 26. Martinson, O.B. and Cole, E.T. (2002) Improving accounting education through outcomes assessment, Management Accounting Quarterly, Vol. 3, No. 2, pp.1 6. Moore, M.G. and Kearsley, G. (1996) Distance Education: A Systems View, Belmont, CA: Wadsworth. Muraki, E., Hombo, C.M. and Lee, Y-W. (2000) Equating and linking of performance assessments, Applied Psychological Measurement, Vol. 24, pp.325 337. Overby, S. (2006) How to hook the talent you need, CIO, Vol. 19, No. 22, pp.40 54. Phillips, V. (unknown) Online degrees public acceptance, http://www.geteducated.com/surveys/ publicacct.asp (retrieved 21 January 2008). Rob, M. (2003) On site: the rise and fall of the e-commerce program, Communications of the ACM, Vol. 46, No. 3, pp.25 26. Rossiter, D. and Crock, M. (2006) Embedding e-learning: a new perspective on change and innovation, Int. J. Learning Technology, Vol. 2, No. 4, pp.279 293. Rovai, A.P. (2001) Building classroom community at a distance: a case study, Educational Technology Research & Development, Vol. 49, No. 4, pp.33 48. Rungtusanatham, M., Ellram, L.M., Siferd, S.P. and Salik, S. (2004) Toward a typology of business education in the internet age, Decision Sciences Journal of Innovative Education, Vol. 2, No. 2, pp.101 120. Shaftel, J. and Shaftel, T.L. (2007) Educational assessment and the AACSB, Issues in Accounting Education, Vol. 22, No. 2, pp.215 232. Shore, B. and Briggs, W. (2007) Competitive analysis of MIS in the MBA core: are trends putting pressure on the MIS course?, Journal of Information Systems Education, Vol. 18, No. 1, pp.63 68. Singh, H. and Reed, C. (2001) Achieving success with blended learning, White paper, Centra Software. Trapnell, J.E. (2005) Foreword, in K. Martell and T. Calderon (Eds.) Assessment in the Disciplines, Vol. 1, No. 1, Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way, Tallahassee, FL: Association for Institutional Research, Florida State University. Vonderwell, S., Liang, X. and Alderman, K. (2007) Asynchronous discussions and assessment in online learning, Journal of Research on Technology in Education, Vol. 39, No. 3, pp.309 328. Zammuto, R.F. (2008) Accreditation and the globalization of business, Academy of Management Learning & Education, Vol. 7, No. 2, pp.256 268. Note 1 Macromedia Inc., Breeze Meeting Resource, http://www.adobe.com/resources/breeze/ meeting/ (retrieved May 2006).