ICEM 1999 1 Principles of Pedagogy and Evaluation for Web-Based Learning Dr. Charalambos Vrasidas Dr. Marina S. McIsaac ICEM 1999, Slovenia Early draft of the manuscript: Vrasidas, C., & McIsaac, M. (2000). Principles of pedagogy and evaluation of Webbased learning, Educational Media International, 37(2), 105-111 Abstract The authors have designed a number of courses at Arizona State University, which were delivered on the Internet and supported with computer conferencing software. In this paper the authors will present an online course, discuss issues relating to online learning environments, reflect on their experiences from teaching online, and discuss the findings of the course evaluation. Emphasis of the discussion will be on the pedagogical principles for effective teaching and evaluation of learning in an online environment. Introduction Few would contest that the use of electronic networks and the Internet in education shape in new ways how we interact, share information, and communicate with each other. During the last four years the authors have designed a number of courses at Arizona State University, which were delivered on the World-Wide-Web and supported with computer conferencing software. In this paper we will present an online course, reflect on our experiences from teaching online, discuss principles of pedagogy and evaluation of web-based learning, and provide suggestions for educators interested in designing their own online course. When talking about online courses we are not referring to courses that have a syllabus online and meet regularly face-to-face. Online courses are those courses that are completely online or have several scheduled online meetings. There might be some faceto-face meetings but the majority of instruction takes place online. Online courses that use extensively text-based conferencing systems have both advantages and disadvantages. Specific examples of advantages and disadvantages of computer conferencing will be discussed as they relate to our experience. One of the biggest advantages of online courses is the fact that they allow for interaction in both synchronous and asynchronous modes. That is, they allow for place and time independence. People can interact at their own time and have the flexibility to take courses without physically walking in the classroom. Because of its synchronous and asynchronous nature, computer-mediated communication (CMC) allows for selfpaced learning and reflection. The real strength of CMC lies on the premises of constructivist epistemology.
ICEM 1999 2 According to constructivism, knowledge does not exist external to the learner. Rather, individual learners construct their own meanings based on their prior experiences. Learning is a result of construction, collaboration, reflection, and negotiation within a rich context in which learning is situated (Brown, Collins, & Duguid, 1989). Jonassen, Davidson, Collins, Campbell, and Haag (1995) argued that the four major system attributes on which the design of constructivist learning environments are based are context, construction of knowledge, collaboration, and conversation. Knowledge building results when learners interact with their peers, collaborate, discuss their positions, form arguments, reevaluate their initial positions, and negotiate meaning. The teacher has become more of a coach on the side rather than the main source for delivering information. In a more decentralized environment such as those facilitated with CMC, students take control of the learning environment, pose questions, moderate discussions, and summarize results. One of the major disadvantages of text-based CMC is the lack of visual and audible cues during communication. Body language, facial expressions, gestures, and voice intonation are all excluded from such an environment. A simple face expression can often communicate so much more than any text message. The same expression said with two different voices, can also have different meanings. This lack of richness of communication in cues communicated is one of the major disadvantages of CMC. Current developments in computer conferencing systems are addressing this issue by allowing the exchange of textual, audible, and visual cues. Like in all distance education courses, online courses require self-motivated students in order to be successful (McIsaac & Gunawardena, 1996). In addition, structure is even more important in online courses to promote interaction and create a successful learning environment (Vrasidas & McIsaac, in press). Furthermore, the technologies involved in distance education require an established infrastructure for an educational institution in order to be able to use it effectively. Such an infrastructure can often be expensive and beyond the budget limits of several institutions. Finally, educators are often intimidated by the technology because they lack the skills and knowledge necessary for developing and managing online environments. Design and development For the design, delivery, and evaluation of several online courses, we developed a process that guided us throughout project completion. The process we followed for designing the online courses was based on several theoretical instructional design models as well as on other factors involved during the development of online instruction. Such factors included user interface design, branching and interactivity, structuring of the information, navigation tools, strategies for promoting online interaction, and distance education research (Berge, 1995; Dick & Carey, 1996; McIsaac & Gunawardena, 1996; Park & Hannafin, 1993; Schwier & Misanchuk, 1993; Smith & Ragan, 1993). It is very important to note that in an online setting there are no prescriptions. Like in all instructional settings, every online course is unique, depending on its goal, content, audience, budget, and the like. What is discussed in this paper is something that
ICEM 1999 3 relates to our experience from which readers can learn and adapt to their own situations. The following discussion will be about an online course we developed, taught, and evaluated during Fall of 1997. The title of the course was EMC503 Current Issues and Problems in Educational Media and Computers and combined both face-to-face and online instruction. Before and during development we asked a few key questions which we will discuss in the following sections. Why develop an online course? Why is the course offered online rather than face-to-face? Often times faculty in colleges and universities are required to develop online courses or adapt some of their traditional courses to be delivered online. In our case, we wanted to reach non-traditional students, because in our courses we have a lot of students coming from the college of Extended Education, who cannot attend the traditional face-to-face classroom. So, the time and place independence that the online environment provides was a big factor in this situation. The course evaluations, in which students gave us feedback about the course, indicated that the flexibility of the online environment was one of the major reasons that led several students to enroll in the course. Which is the audience and what are the learner characteristics? In any instructional setting, the audience is crucial for the design, implementation, and success of educational programs. The target audience of the course is a significant factor that will influence the decision as whether to deliver the course online or face-toface. Bellow are examples of some of the questions that we were interested in answering before the development of the course. Do the learners have access to microcomputers and the Internet? Are the learners familiar with the use of microcomputers? How experienced are they in navigating the World-Wide-Web, using electronic mail, engaging in computer conferencing, uploading downloading files, and the like? What do the learners already know about the subject matter of the course? What are their attitudes towards the subject and the delivery medium? We did not have access to all this information beforehand. Therefore, we used the first three face-to-face meetings at the beginning of the course to get this information and help every student with the use of conferencing software. What are the limitations? What are the resources available for the development of the course, what is the budget for the course, and what is the timeline for development? Who will work on the project and what are the skills of the staff in developing web-sites, installing and running conferencing software, and administering servers? Developing and delivering an online course requires teamwork. Possible members in a team environment are the content expert, instructional designer, and web developer. It is also very important to have a facilitator. In our case we had a facilitator, and also a
ICEM 1999 4 graduate student who was doing an internship with us and was responsible for helping students when they had problems with technology. As far as the technology is concerned, we already had access to the computer conferencing system FirstClass that was widely used in our College. What is the budget for developing the on-line course? How much of that budget will be applied for the development of the course and to buy licenses for FirstClass? The limited bandwidth will also determine the general look of the course s web-site (limited use of video, sounds, animations, images). Such a project will require much more time for planning and development. What is the content? Some contents are more appropriate than others. For example, subjects that allow for debate, discussion, and exchanging ideas are very appropriate for online courses. The topic of issues and problems in educational media consisted of several controversial topics for which students had to take positions, structure their arguments, and defend them. In addition, we found particularly helpful to develop a detailed content outline, which illustrated the structure of the content in a meaningful way. For developing the course s website, a detailed content analysis is very important for modularizing the content into small chunks. This step is crucial in determining the nodes and links of the site, branching and navigation, options provided, and the design of the user interface, as they will be made available on the web. What are the goals/objectives of the course? The goal of this course was to provide a general overview of the field of education to the graduate students of our program. The specific objectives were the following: 1. Identify and describe at least three current problems or issues in educational media or educational technology. 2. Write annotated summaries from articles found in different professional or scholarly journals in this field. 3. Write a research paper examining in detail one issue or problem in our field. How will the objectives be measured? The evaluation of student performance was based on a variety of methods. Because of the nature of the online environment, and the lack of non-verbal cues that often provide a lot of valuable information to the teacher, multiple methods for collecting data about student learning were employed. Evaluation data were collected using weekly assignments, messages posted in the weekly discussions, observation of discussion moderation by students, a midterm examination, a final research paper, and a final presentation. It is important for online courses to have multiple ways of measuring student learning and evaluating the effectiveness of distance education. Multiple sources of data are more likely to provide a more complete picture of what is happening in an online course.
ICEM 1999 5 What are the strategies and design decisions? What are the instructional strategies that will be employed in order to meet the objectives? How will the information be presented? What are the pedagogical principles that should guide the design and structure of online courses? Some of the design considerations for designing online instruction are: Selection of online and off-line activities, Decisions about how much of the content will be online or when the learner will be referred to other resources (printed readings, videos, CD-ROMs, etc.), Issues relating to promoting student participation, and moderating online discussions. In the following section we will describe the course in more detail and which will illustrate the strategies we employed and pedagogic principles that guided our work. Course description The course EMC-503 Current Issues and Problems in Educational Media and Computers combined both face-to-face and online instruction. The course was offered through the department of Educational Media and Computers and the College of Extended Education at Arizona State University. Arizona State University is delivering a variety of distance education courses. Some of these courses are offered via two-way video twoway audio, and others are offered on the Web and supported with computer conferencing. The content of the course consisted of issues and topics such as instructional technology, distance education, media and learning, learning theory, communication theory, and equity. The class was supported with the computer conferencing software FirstClass and a website that had information about assignments, class schedule, resources, and postings of students work. FirstClass is an advanced text-based computer conferencing system that allows for synchronous and asynchronous communication. It integrates electronic conferencing, electronic mail, access to information databases, file transfer and sharing, and real time online discussions among multiple users. The course was a combination of face-to-face instruction and online interaction. Class met face-to-face during the first three weeks during which students were introduced to the format, requirements, and schedule of the course. In addition, students were guided through the process of downloading and installing FirstClass software and accessing the class information online. Instructors and students posted a 250 word autobiographical memo on an online conference for class to read. After the first three face-to-face meetings, class continued online for nine weeks. During the last three weeks class met face-to-face for students presentations of their final projects. Two students that could not attend the face-to-face meetings made arrangements and presented their work remotely via computer-conferencing. The assignments and requirements of the course were made very specific with regards to length, deadlines for submission, and grading policies. Once the assignments and papers were completed, they were posted on the web and made available for everyone in the class to read. General issues about the course were posted in the News
ICEM 1999 6 folder, an online conference within FirstClass where students and instructors could post messages and announcements regarding the course. Contributions to weekly discussions and assignments were also submitted using FirstClass. In addition to the weekly readings and reflection papers, each week a student was assigned as a moderator and she had to post two questions for discussion on the weekly topic. One question was posted on Monday and one on Wednesday. Every student had to respond to each question at least once during the week. On Sunday of each week, the moderator was responsible to summarize the results of the discussion and post them online. Reflections on our experience and suggestions for pedagogy and evaluation In this section we will reflect on our experience from teaching online and discuss the aspects of the course that worked and those that need improvement. We will also provide some simple suggestions for pedagogy and evaluation of online environments. A questionnaire administered during the middle of the course indicated that learners were satisfied with some aspects of the course and unsatisfied with some others. Below are the key issues that came out the evaluation at the middle and at the end of the course. These issues that are briefly discussed below are not rules to be followed, but suggestions that are based on our experiences. Always allow enough time and plan ahead Allow plenty of time for planning, development, and revisions. Changes to the content require changes of the web-site and maybe of the FirstClass conferencing structure. During the development of online courses revisions are always necessary. Therefore, allow plenty of time to plan and develop the course. Furthermore, there is the false assumption among educators that teaching online is easy and it doesn t require as much time as traditional instruction. Our experience tells us that online teaching requires much more work. In fact, online courses require more detail planning and structuring to successfully engage students in a valuable learning experience. Do not minimize the front-end analysis phase Spend enough time addressing some of the questions discussed in an earlier section. It is very helpful and it will save you a lot of time in the long run. For the website, draw out in detail the content outline. This outline will help in breaking down the content into small chunks and allow quick reviews and revisions of the material. A flowchart and timeline are good tools to use to keep all members of the team working in the same direction and at the same pace. Provide immediate feedback Feedback has been associated with kinds of responses that provide information to students about the correctness of their assignments, homework, and class contributions. In distance education feedback is more important than just a mechanism of informing the student on how well she did on an assignment. In face-to-face situations nonverbal
ICEM 1999 7 gestures are constantly exchanged thus providing both the teacher and learners with feedback. A verbal comment by the teacher, a smile, a facial expression, and a gesture, are all ways with which students can get feedback on their work and ideas. A confused face can indicate to the teacher that she needs to elaborate more on a topic. In the online environment, however, you loose all the contextual cues of communication, which are important in creating the feeling of social presence. Frequent feedback is very important in online courses. Students need many opportunities for feedback on their assignments, discussion participation, and overall progress. Feedback needs to be personalized and addressed to the individual student s work. General feedback addressed to the class as a group is also advisable, but it is individual feedback that touches the student. In addition, it is important to contact the students on a weekly basis to check if they are having any problems with the course, assignments, use of technology, and get their continuous feedback for improving the course. Participate in the discussions and provide enough modeling The participation of the instructors in online discussions moderated by students adds more credibility to the discussion. Students wanted to hear from the instructors and asked them to participate in the online discussions more often. They felt that the instructors were the experts and added more credibility to their discussions. A student stated in the course evaluations: When the teacher posts a message, the whole discussion becomes instantly valid. Therefore, instructors can participate in the weekly discussions, comment on the student s contributions, and add comments and references that relate to the weekly topic. For example, in a discussion on constructivism the instructor can contribute a comment on a student s posting and submit an Internet address where related material can be found online. However, this needs to be done very carefully. The instructor should not be seen as an authoritarian figure, but rather as a coach and a facilitator and refrain from imposing her views on the discussion but carefully guide the students in exploring an issue through multiple perspectives. While participating in the discussions, the instructor has the opportunity to model expert behavior and appropriate etiquette. The first discussions should be moderated by the instructor so that students can have concrete examples of how to go about structuring and moderating their own discussions. We found particularly helpful assigning as first required reading Paulsen s (1995) chapter dealing with the moderation of online discussions and guidelines for moderators. Promote interaction and social presence Several studies indicated that social presence could be promoted in a CMC setting by employing strategies that promote interaction (Gunawardena, 1995; Vrasidas & McIsaac, in press). Because of the lack of audio-visual cues in text-based CMC settings, users invent other means for compensating for the lack of those cues. It is in the hand of the instructor and the moderator to create an environment within which learners can feel socially present. We found that students did not interact among themselves as much as
ICEM 1999 8 we have expected. We did not have any collaborative activities designed and therefore students did not collaborate with each other. One technique that can be used is the structuring of collaborative learning activities. This came as a suggestion by four students during the course evaluations. CMC allows the use of techniques such as collaborative group work, group discussions, and brainstorming. Learners on the same site or at distant sites separated by space and or time, can interact and exchange ideas through computer networks. In order to increase the feeling of social presence and the idea of online learning communities, group activities need to be planned in advance. Organizing more group activities can increase learner-learner interaction and take advantage of the medium that can support collaborative work. In addition to assigning pairs for moderating online discussions, some other kinds of activities that can be incorporated to increase learnerlearner interaction include group assignments, group projects, and online group debates. Online environments that rely heavily on text-based communication lack the visual and audible cues present in traditional face-to-face classrooms. As a result, communication that takes place in such environments has often been criticized as lacking in richness. We feel that it is in the hands of the instructor to create the feelings of the learning communities and social presence. Only good balance between the structure of the course and the online interactions can provide for an optimal learning experience. Evaluation of online learning The online classroom, more than in the traditional classroom, requires a variety of methods for evaluating student learning (Hiltz, 1990). In the traditional classroom there are several ways that the teacher can use to evaluate students. A confused face or a head nodding can communicate a lot to the teacher. Non-verbal communication is a big part of the evaluation process of the traditional face-to-face classroom. In the online, the teacher can only evaluate what she has access to. She doesn t have access to facial expressions, voice intonation, or body language. Therefore, a variety of methods are essential for evaluating students and educational programs delivered online. In our case, for evaluating students and the course we gathered information from students weekly assignments, students moderations of online discussions, students postings in online conferences, students final papers and presentations, the midterm evaluation, face-to-face discussions we had with students, and our personal reflections on the course. Conclusion Teaching online was a great learning experience. Reflecting on ones work can help improve pedagogical practices in the field of online education. Our experiences illustrate that teaching online is not as easy as it may sound. It requires detailed planning and hard work. There is a strong need for more empirical studies to be conducted that can provide guidelines for structuring and evaluating online environments. The issues addressed in this paper illustrate that structure is crucial for promoting interaction and social presence in distance education. Interaction among students and interaction between students and instructor needs to be carefully planned. The online nature of the course described
ICEM 1999 9 illustrates that multiple methods for evaluation are essential in order to get a clear picture of what is happening in online environments. References Berge, Z. L. (1995). Facilitating computer conferencing: Recommendations from the field. Educational Technology, 35(1), 22-30. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42. Dick, W. & Carey, L. M. (1996). The systematic design of instruction (4th ed.), New York, NY: Harper Collins College Publishers. Hiltz, S. R. (1990). Evaluating the virtual classroom. In L. M. Harasim, (Ed.), Online education: Perspectives on a new environment (pp. 133-184). New York: Praeger. Jonassen, D., Davidson, A., Collins, M., Campbell, J., & Haag, B. B. (1995). Constructivism and computer-mediated communication in distance education. The American Journal of Distance Education, 9(2), 7-26. Gunawardena, C. N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1(2/3), 147-166. Mason, R. (1994). Using communications media in open and flexible learning. London: Kogan Page. McIsaac, M. S., & Gunawardena, C. N. (1996). Distance education. In D. H. Jonassen, (Ed.), Handbook of research for educational communications and technology (pp. 403-437). New York: Simon & Shuster Macmillan. Park, I. & Hannafin, M. J. (1993). Empirically-based guidelines for the design of interactive multimedia. Education, Technology, Research and Development, 41, (3), 63-85. Paulsen, M. F. (1995). Moderating educational computer conferences. In Z. L. Berge, & M. P. Collins, (Eds.), Computer-mediated communication and the online classroom (pp. 81-104). Cresskill, NJ: Hampton Press, Inc. Schwier, R. A. & Misanchuk, E. R (1993). Interactive multimedia instruction. Englewood Cliffs, NJ: Educational Technology Publications. Smith, P. L. & Ragan, T. J. (1993). Instructional Design. New York, NY: Macmillan Publishing Company. Vrasidas, C. & McIsaac, S. M. (in press). Factors influencing interaction in an online course. The American Journal of Distance Education.