Improving the educational process by joining SCORM with adaptivity: the case of ProPer

Transcription

1 Int. J. Technology Enhanced Learning, Vol. 4, Nos. 3/4, Improving the educational process by joining SCORM with adaptivity: the case of ProPer Ioannis Kazanidis* Kavala Institute of Technology, Agios Loukas, 65404, Kavala, Greece *Corresponding author Maya Satratzemi University of Macedonia, 54006, Thessaloniki, Greece Abstract: Over the last several years, Learning Management Systems (LMSs) have adopted technological standards such as Shareable Content Object Reference Model (SCORM), in order to achieve interoperability and reusability of their educational content which still remains static. On the other hand, adaptive hypermedia have adapted educational material to learners characteristics and preferences so as to help them with their study, however, it is difficult for their content to be reused among different systems. This paper discusses a way to improve the educational process both for learners and instructors by joining SCORM compliant LMSs with adaptivity. It proposes a model for e-learning systems construction and presents ProPer, a system that combines all the popular adaptive technologies with LMS functionalities and the adoption of the SCORM specifications. The evaluation results indicate that ProPer is an easy and useful adaptive system and helps students achieve a better learning outcome in a shorter study period. Keywords: adaptive hypermedia; SCORM; shareable content object reference model; learning styles, AEHS; adaptive educational hypermedia systems; technology-enhanced learning. Reference to this paper should be made as follows: Kazanidis, I. and Satratzemi, M. (2012) Improving the educational process by joining SCORM with adaptivity: the case of ProPer, Int. J. Technology Enhanced Learning, Vol. 4, Nos. 3/4, pp Biographical notes: Ioannis Kazanidis is an Adjoined Assistant Professor at the Kavala Institute of Technology, Greece. He received the MSc degree in Computing from the Coventry University, and his PhD degree in Educational Technology and Adaptive Educational Systems at University of Macedonia. He has an extensive experience in lifelong learning, and helping adult groups. His main research interests lie in the area of adaptive systems, data mining methods and algorithms, virtual words in education and computer-supported collaborative learning. He has published more than 15 articles in international journals and book chapters, and 25 papers in proceedings of conferences. Maya Satratzemi was awarded the BS degree in Mathematics from the Aristotle University of Thessaloniki in 1980, and the PhD degree in Copyright 2012 Inderscience Enterprises Ltd.

2 232 I. Kazanidis and M. Satratzemi Informatics (Algorithmic Graph Theory) in She is a Professor at the Department of Applied Informatics, University of Macedonia. Her main research interests lie in the area of Programming Languages, Educational software, Didactics of Informatics, Algorithms, Algorithmic Graph Theory. She has published more than 30 papers in journals and 70 papers in conferences proceedings. 1 Introduction Over the last several years, Learning Management Systems (LMSs) have adopted technological standards such as the Shareable Content Object Reference Model (SCORM) in order to achieve interoperability and reusability of their educational content. On the other hand, Adaptive Educational Hypermedia Systems (AEHSs) have adapted educational material to learners characteristics and preferences so as to help them with their study. Until now, most AEHSs do not adopt common technological standards and therefore, their educational content cannot be reused. While, on the other hand, LMS content remains static. The goal of the current research is to improve the e-learning process for both learners and teachers. In order to achieve this, a development model has been proposed and an adaptive and adaptable educational system has been developed. This system provides adaptive courses according to learner characteristics, progress and learning style, while at the same time, it lets instructors reuse and exploit educational content from other courses adopting SCORM specifications. In addition, SCORM compliant educational material has been designed for the system s evaluation. The paper is organised as follows: In section 2, the proposed e-learning systems construction model is presented, and in section 3 the background framework is briefly described. Section 4 describes the related work and section 5 presents the prototype and its functionality. The evaluation results are reported in section 6. Finally, section 7 summarises and presents conclusions along with future work. 2 Proposed model This work proposes a model for more efficient online courses and e-learning systems that comes as a solution to the problems described above. According to the proposed model, an e-learning system should follow three general rules. First, it has to provide LMSs functionalities. The range of the functionalities that LMSs provide make them popular with end users and has helped their spread. Therefore, LMS functionalities are necessary for an online system. The second rule proposed by many researchers (Stern et al., 1997; Pilar da Silva et al., 1998; Kavcic 1999) in order to avoid the major problems of e-learning courses is the application of adaptivity and interactivity of online instruction. This rule emphasises educational systems with personalised instruction to user needs, preferences, and characteristics, adapting user navigation and content presentation. In addition, for better learning outcomes and according to Papanikolaou et al. (2003) and Triantafillou et al. (2003), adaptation should take user learning style into account.

3 Improving the educational process by joining SCORM with adaptivity 233 The third rule of the proposed model concerns the adoption of a widely accepted technological standard. SCORM is the preferred choice, since most LMSs use it. It is a well-known fact that the educational content used in an AEHS is most times not reusable in any other, which means that educators have to do a lot of added work in designing and developing their own educational material. We strongly believe that it would be a great impact on the growth of AEHSs, if educators could reuse other people s qualitative educational material that has been successfully applied. The adoption of the proposed model led to the implementation of a SCORM compliant LMS with adaptive and adaptable features named ProPer. 3 Background framework This section describes the background framework for the implementation of the proposed model. First, AEHS architecture and applied adaptation technologies are discussed. Since the proposed model for e-learning systems recommends the adaptation to user learning style, a special reference is made to various learning style models and in particular to the Honey and Mumford model, which was used for the pilot courses. The section finishes with a presentation of the main components of SCORM. 3.1 AEHS The architecture of all AEHSs follows a general structure consisting of three main components. These are (i) the Domain Model (DM) which represents the system s domain knowledge, (ii) the User Model (UM) which represents the learner s knowledge of the domain and (iii) the Adaptation Module (AM) which defines the rules of system s adaptation. Various technologies have been used in order for AEHSs to provide adaptivity. Brusilovsky (1998) distinguished these techniques into two main categories according to the kind of adaptation applied: the Adaptive Presentation (AP) and the Adapted Navigation (AN). 3.2 Learning styles According to Honey and Mumford (1992) learning style refers to a person s habits and patterns of behaviour that determine the desired means of learning. Some of the learning style models used by several AEHSs are: Kolb s (1984) experimental learning model; Honey and Mumford (1992) based on Kolb s model; the Felder and Silverman (1988) model; the Witkin s Field Dependent/Field Independent model (Witkin et al., 1977) and Gardner s (1993) theory of Multiple Intelligences (MI). The system we propose supports the construction of courses that provide adaptation potentially consistent with many of the learning style models since it lets authors define personalisation through appropriate JavaScript. However, the implemented prototype makes use of the Honey and Mumford categorisation since it is one of the most widely accepted models and already has been used by other related AEHS such as INSPIRE and AHA. According to the Honey and Mumford model (Honey and Mumford, 1992), learning is a process of knowledge construction through four distinct stages of a cycle: (a) having

4 234 I. Kazanidis and M. Satratzemi an experience; (b) reviewing the experience; (c) concluding from the experience and (d) planning the next step. The student can start from any point in the cycle and progress to the others. Each stage is related to a particular learning style. Thus, the corresponding learning styles are the: (a) activist, (b) reflector, (c) theorist and (d) pragmatist. 3.3 SCORM The main aim of SCORM (ADL, 2009), known as the Shareable Content Object Reference Model, is to offer courses that are reusable, easily accessible, interoperable to many systems and platforms and durable to possible system software updates. There are three main components of SCORM technical specifications: the Content Aggregation Model (CAM), the Run-Time Environment (RTE) and the Simple Sequencing and Navigation (SSN) specification. The learning content in SCORM courses consists of Sharable Content Objects (SCOs).The SCOs communicate with the LMS via an API in order to retrieve or store data. 4 Related work In literature and praxis, there are numerous AEHSs and SCORM compliant LMSs. However, there are not many systems that combine these two frameworks. This section gives a short review on chosen related work with similar functionalities and tools. The chief feature of our system is the combination of LMS technologies and SCORM API with adaptive hypermedia technologies. This is its main advantage over identical systems. Even though, many popular LMSs that support SCORM courses (Moodle, Claroline, Web-CT) do not personalise the learning process to each user. On the other hand, there is a stream of research that tries to join LMSs and AEHSs using certain emerging standards, like SCORM (Brusilovsky, 2004). Some such systems are OPAL (Conlan et al., 2002) and VIBORA (Morales, 2003) which both adopt the SCORM standard. AdeLE (Gütl and Mödritscher, 2005) is another system which can be connected to LMSs with slight modifications. Some other systems like WINDS (Kravcik and Specht, 2004) are structured by SCORM compliant learning objects even if they do not support native SCORM compliant courses. There are also a number of widely known AEHSs, such as ELM-ART, AHA!, INSPIRE, AES-CS, CSC383. ELM-ART (Brusilovsky et al., 1996) supports adaptive navigation techniques. The newer version of AHA! (Romero et al., 2005) supports the import of SCORM compliant courses and provides adaptation according to user learning style (Stash et al., 2004) implementing the Honey and Mumford Model. There are also some other adaptive systems that provide adaptation according to the user s learning style such as INSPIRE (Papanikolaou et al., 2003), AES-CS (Triantafillou et al., 2003), CSC383 (Carver et al., 1996), TANGOW (Carro et al., 1999), WINDS, EDUCE (Kelly and Tangney, 2006) and WELSA (Popescu, 2008). All of them adopt a learning style theory and apply it through content personalisation. As shown in Table 1, there is not one system that implements all the characteristics listed below. We believe that a possible solution to the major problems of web-based educational systems, as stated in (Kazanidis and Satratzemi, 2007, 2009) and a reliable

5 Improving the educational process by joining SCORM with adaptivity 235 answer to the needs for educational content reusability and interoperability would be the conjunction of AEHSs with LMSs facilities and SCORM adoption. We proposed the development of a SCORM compliant LMS with adaptive and adaptable capabilities which would also support adaptation to user learning style, progress in the course, educational goals and preknowledge of the domain. In the following paragraphs we present a prototype, called ProPer, along with a formative and summative evaluation. Table 1 Related work and proposed system characteristics System Adaptive courses Learning styles SCORM courses LMS Moodle Claroline WebCT OPAL VIBORA AdeLE WINDS Felder & Silverman ELM-ART Supports SCORM Learning objects AHA! Honey & Mumford Import INSPIRE Honey & Mumford AES-CS Field Dependent/Field Independent TANGOW Felder & Silverman CS383 Felder & Silverman EDUCE MI theory WELSA Proposed System Unified Learning Style Model Can be connected to LMSs Honey & Mumford 5 ProPer ProPer is an integrated adaptive and adaptable learning environment that offers LMS administration facilities and conforms to SCORM specifications. It comprises a combination of an AEHS and an LMS, while it provides a variety of additional educational features both for the student and the teacher. 5.1 System architecture and implementation ProPer s architecture is a combined architecture of a SCORM LMS and an AEHS. It adopts the structure of a typical SCORM Run Time Environment (RTE) with the addition of an adaptation module and the extension of the preexistent Domain and User Models. In brief, ProPer encompasses six main modules (Figure 1):

6 236 I. Kazanidis and M. Satratzemi The Domain Model (DM) represents the domain knowledge of the system. The User Model (UM) that represents the particular user s knowledge of the domain as well as his/her individual characteristics. ProPer uses a multilayered overlay UM which stores three types of data: (i) data about user knowledge, (ii) data about user actions and goals, and (iii) domain independent data. The Adaptation Module (AM) which interacts with the DM and UM in order to provide the system s adaptive functionality. It incorporates a set of adaptation rules which define the way personalisation is applied. The RTE sequencer, which interacts with the DM and delivers the appropriate educational content to the learner. User Tracker (UT) which monitors the learner s interaction with the system and stores all the essential data into the UM. Feedback visualiser which initially calculates the feedback information and then visualises the results and delivers them to the user. Figure 1 System s architecture (see online version for colours) The system retrieves course files initially from a zip file, which contains a manifest XML file and the entire html and media required files. The DM data is stored in Java Object Files. Additional data about the course, however, is stored in the database. UM data is stored both in separate Java Object Files and in the database. The systems database (Figure2) is consisted by 14 tables. Each time a user visits a SCO, the tables itemuserinfo and the usercourseinfo are updated according to user behaviour. More specifically from table itemuserinfo the system updates the user Score, totaltime (time the user spent studying the SCO) and NumHitAttempts (is appending by one on every visit). At the same time the percentage of user score for the entire course and the user score according only to his/her educational goals are calculated and stored to userscore and usertargetscore columns, respectively. User domain independent data is stored in the table userinfo.

7 Improving the educational process by joining SCORM with adaptivity 237 Figure 2 System s database (see online version for colours) For the system interface a multi-frame web page is used which is divided into four main frames (Figure 3).

8 238 I. Kazanidis and M. Satratzemi Figure 3 System s interface (see online version for colours) On the left hand side, the system visualises the course structure. On the bottom left of the screen, the user s overall score and a score according to his/her educational goals is presented. Moreover, a button named Next leads to direct guidance navigation. Finally, there are two more buttons ( Learned and Not Learned ) which let the user explicitly define whether or not s/he has learned the current activity. With this feature users can adapt their UM and amend the system s estimation about their knowledge. On the top of the screen, there is a functionality toolbar with all the functions and tools that the system provides. The most important tools among others are the User Model and Java Tool. User Model lets the user define, his/her educational goals and preknowledge on the domain. In addition s/he can select some sets of goals predefined by the instructor, designed for particular categories of users (Figure 4). The Java Tool is a simple Java editor and compiler module which was added at the last version of ProPer since it was used for teaching object oriented programming with Java. Java Tool lets user write Java code and submit it for compilation and execution (Figure 5). The innovative feature of Java Tool is that it can assess the user through the generated compilation and execution output. The assessment algorithm marks the learner with 50% if the Java program is compiled successfully. The rest 50% involves the program output. In particular instructors define both input and output values for program testing. In the case execution output is correct learner is assessed with 100% for the current program. The learner s grade, the time spent on program development, the number of compilations and the different program versions between compilations are available to instructor for more precise student assessment.

9 Improving the educational process by joining SCORM with adaptivity 239 All the above information along with the other student model data are analysed and the system provides detailed and visualised feedback to both students and authors, enabled them to find possible weaknesses and problems arise during the educational process. Figure 4 User model tool interface (see online version for colours) Figure 5 Java programming tool interface (see online version for colours)

10 240 I. Kazanidis and M. Satratzemi The main frame of the system interface displays the educational material of the course. The presentation of this material can be either statically or dynamically created according to the system UM. An instance of adaptive content presentation is presented later in the paper. 5.1 User model ProPer stores data about user personal characteristics, progress and learning style into the User Model. UM is updated in three cases: each time the user visits a new SCO, when s/he abandons a SCO and each time user manually defining his/her educational goals and pre-knowledge on the course domain. In order to succeed this, appropriate database was developed. A problem that arises is how a SCO may change user model in system s database and how it can estimate if its content is considered as known by the user or not. In order to overcome the above problem, the following procedure was applied. According to SCORM, each SCO includes an attribute called minnormalizedmeasure which is the lowest score that user has to succeed in order the SCO to be considered as known. When the user leaves from a SCO, it marks the user through the command SetValue( cmi.score.scaled, userscore ) where userscore is the score that SCOs estimate for the user. If user score is higher than the score into the minnormalizedmeasure attribute then the SCO is considered as known and appropriate adaptation is applied as it is stated in the next section. 5.2 Adaptive strategies ProPer supports several different adaptive strategies. The simplest strategy provides sequential navigation. In addition direct guidance is provided by the button Next on the system s interface and delivers the most appropriate material for study on the screen. Link hiding, disabling and removal techniques can be applicable using the course s SCORM functionality. Moreover, ProPer provides adaptive annotation of the TOC links (Figure 6). The TOC is constituted by links to course units and folders. The folders are displayed with a folder image and their titles, while the course units are linked with an image of a book as well as their title. The system annotates the links as follows according to the values of the table itemuserinfo, into the system database: opened book: the user has already visited an activity. The column NumHitAttempts from the table itemuserinfo is more that 0; a green is displayed on the book: the corresponding activity is considered by the system as known. Score column from itemuserinfo table is higher than the minnormalizedmeasure of current SCO; green book: according to the UM this seems to be the most appropriate activity for study. All the previous activities are considered either known (score>=minnormalizedmeasure or preknowledge=true) or they do not belong to user educational goals (target = false); red book: this activity is not recommended for study because it is beyond the user s educational goals. Column target of itemuserinfo is set to false;

11 Improving the educational process by joining SCORM with adaptivity 241 blue book: the activity is one of the user s goals but it is not proposed for study. The column target of itemuserinfo is set to true but a previous activity which belongs to user educational goals is considered as uknown (score<minnormalizedmeasure and preknowledge=false); gold book: this is the current activity. Figure 6 Table of contents and symbols explanation (see online version for colours) ProPer also can deliver courses that adapt content presentation to user learning style according to Honey and Mumford model (1992). System initially acquires user learning style through the Learning Style Questionnaire which proposed by Honey and Mumford (2006) and modifies student model. Afterwards it adapts the content presentation, through appropriate JavaScript, according to the pre-discovered user learning style. The adaptation strategy which is followed provides all learners with the same Knowledge Modules (KMs). However, the various KMs sequence is adapted according to user learning style and KMs are either embed in the page or appeared as links. In order to achieve this kind of content adaptation, courses should follow a specific format. In particular, content is distinguished into three different levels of performance: Remember, Use and Find according to Merrill (1983) Component Display Theory (CDT). CDT associates each performance level with a different combination of Primary Presentation Forms. According to the proposed design, the various KMs of the outcome concepts (questions, theory presentations, examples, exercises, activities, etc.) constitute different instructional primitives (Van Marcke, 1992) which are joined together in various ways. Thus each SCO of the course is consisted by a number of different KMs such as Theory, Example, Question, Activity, Exercise, etc. In particular for Activists learners, the Use level of performance has to be activityoriented and thus an activity KM is appeared on the top of the page while the rest of the KM are appeared as links in a specified order. For the rest of user learning styles that is Reflectors, Theorists and Pragmatists KMs presentation has to be example, theory and exercise oriented, respectively, and thus the correspondent KMs are initially presented following by the other KMs links as previously discussed. In order to achieve the above instructional strategies, a SCO which contains the Honey and Mumford questionnaire (Honey and Mumford, 1992) was developed. This SCO has to store the results of the questionnaire as a score in a SCORM objective named lstyle. This objective is pre-defined in the manifest file of the course with the appropriate XML code. A snapshot of the XML code that defines the lstyle objective is

12 242 I. Kazanidis and M. Satratzemi presented in Figure 7, while a JavaScript function that stores user learning-style score in the lstyle objective is presented in Figure 8. Therefore, the other course SCOs can read the lstyle objective value and obtain the user learning style. These SCOs include: various knowledge modules, so as to adapt the sequence of their presentation; a mechanism that can communicate with the system and read the lstyle objective; and a mechanism that adapts the knowledge modules presentation sequence according to the user learning style. These two mechanisms were implemented through additional JavaScript code for every SCO of the course. Figure 7 XML code into the manifest file of the course that defines an objective named lstyle (see online version for colours) Figure 8 JavaScript function that stores user learning style score in the lstyle objective (see online version for colours) 6 Evaluation of ProPer In this section of the course construction framework, the system s formative and summative evaluation will be presented. Our evaluation is based on procedures followed in other studies (Samarakou et al., 2006). Therefore, the applied procedure was divided into four main stages: 1 the definition of evaluation hypotheses and educational goals; 2 the formative evaluation; 3 iteration of the system and the course; and finally 4 the summative evaluation in order to evaluate the system and how closely it meets the initial objectives.

13 Improving the educational process by joining SCORM with adaptivity 243 The educational goal of the evaluation was to teach students OOP using Java via an online environment. The evaluation hypotheses were the following: H1. According to the proposed model SCORM, adaptation and LMS functionality may be combined all together into an integrated adaptive educational hypermedia system which is easy and useful for both learners and educators. H2. Authors found useful SCORM as a standard for their courses. H3. The system which implements the proposed model helps educators to manage their courses and find any potential student and/or course weaknesses. H4. Students learn faster when AN according to their UM is applied. H5. Students learn more when AN according to their UM is applied. H6. Students learn more when AP according to their UM is applied. H7. Students learn more when both AN and AP according to their UM is applied. 6.1 Formative evaluation The formative evaluation is an integral part of the design methodology, the results of which are used to make the system more effective and efficient (Triantafillou et al., 2003). We based the formative evaluation experiment on the Technology Acceptance Model (TAM) (Davis, 1989). In accordance with this model, the main factors that affect the actual application of a system are the perceived ease of use and usefulness. Therefore, one of the main goals of the formative evaluation was to evaluate the system usefulness and ease of use and the adoption of SCORM which corresponds to H1 and H2, respectively. More specific formative evaluation goals can be listed as follows: evaluate the usefulness of overall or separate system features; find SCORM usefulness for authors; find any possible effect on the educational process. For the formative evaluation procedure we adopted the Tessmer Model (Tessmer, 1993), which identifies the following four phases: Expert Review One-to-one evaluation Small group Field trial These phases are conducted in the above stated order while revision of the system is carried out on the completion of each phase. The first three phases of the formative evaluation, included system presentation to experiment participants, use of the system for half an hour and a structured interview based on three qualitative questionnaires one for each evaluation phase.

14 244 I. Kazanidis and M. Satratzemi The first expert review show that experts liked system design and adaptivity while they proposed some additions such as to provide a feedback mechanism. They also pointed that it is very promising for educational content quality that the use of SCORM let authors reuse educational content. The one-to-one evaluation was conducted for both educators and students and it helped us acquire their opinion about system easy of use and usefulness. Educators were very satisfied by system functionality and they stated that it can help them manage their classes and find possible students and course weaknesses through the use of feedback visualiser. They also found very useful the adoption of SCORM, however they commented that still there is a lack of an easy and free SCORM authoring tool. The objective of small group evaluation which is followed was to discover any system inconsistencies. During the session a problem with the low Java Heap Memory was determined, which was instantly fixed. Influenced by other studies (Brusilovsky et al., 2004; Truong et al., 2005) and due to the quantity of information that subjects would have to learn, it was decided that the experiment should consist of five two-hour sessions. The number of subjects in accordance with (Papanikolaou et al., 2003; Brusilovsky et al., 2004; Samarakou et al., 2006) was defined at 22 students. First, a preliminary two-hour session was held, which was considered necessary so as to select the students who would participate in the experiment according to their responses on the profile questionnaire and their score on the pre-test that were given. Following the traditional evaluation methods of adaptive learning environments, we divided the subjects into two similar groups. One (group A) worked with ProPer and the other (group B) with the ADL SCORM Runtime Environment which does not provide adaptation. Afterwards a detailed scenario of the experiment was given to the subjects and they start studying the course. The scenario objectives required the students to learn the Java Objects theory which was included in two specific chapters of the course. The same procedure was followed in the next three sessions. It is worth mentioning that two students (one from each group) lost one of the sessions so they were excluded from the experiment. During the sessions ProPer kept records of learners visits, scores, time spent on each activity and navigation preferences. In the fifth session, subjects of group A completed an assessment questionnaire about ProPer. Afterwards both groups took a post-test on Java Objects similar to the pre-test in order to estimate the knowledge gained. The session ended with a short discussion between the researcher and the students of the ProPer group Results During the formative evaluation both quantitative and qualitative data were gathered. The results of the pre-test between the two groups found no significant differences on the prior knowledge of each group. Therefore, both groups were considered equal. Two t-tests with alpha level at 0.05 were computed on the post-test results about user course completion and abstracting pre-test results and on the data collected, in order to check experiment hypothesis H4 and H5. The dependent variables were completion and knowledge gained (post-test results pre-test results) respectively. As far as the post-test results were concerned, the t-test found no significant differences on the knowledge gained by the subjects, meaning that adaptive navigation did not significantly influence students knowledge acquisition. These results did not surprise us for two main reasons: (a) other studies (Brusilovsky, 2003; Papanikolaou et al., 2003) have shown that many

15 Improving the educational process by joining SCORM with adaptivity 245 times, adaptive navigation has no significant effect on the knowledge acquisition of the subjects; (b) the course construction was simple and the scenario guided subjects on a predefined path to the right course units. Papanikolaou et al. (2003) state that this is one reason why the evaluation of adaptive navigation should be avoided. Therefore H5 initially was not confirmed. However, the t-test did find a significant difference (p = 0.001) in course completion between the two groups. More specifically, group A subjects moved faster and were more goal oriented in the course than group B subjects. Although subjects from both groups had studied almost an equal number of units, some in the first group (the most experienced), avoided studying already known units and so managed to move further along in the course. Therefore H4 was confirmed. An assessment attitude questionnaire aimed at determining both the subject s experience in using the system and the usefulness of the system s features was applied. The questionnaire was consisted of 55 questions. Most of them were 5 likert scale agreement questions (1. Totally disagree 5. Totally agree). Additionally, there were five open-ended questions and two multiple-choice questions. The questionnaire results show that students believe that almost all features of the system have high usefulness. As regards the interaction assessment, 90% of users were neutral or agreed that the user model corresponds to real user characteristics. In addition the majority of the participants were satisfied by system adaptation (4 out of 5 in likert scale). The most useful adaptation technique according to participants was the provided adaptive link annotation (4.6). The responses revealed also that a vast 70% of subjects have the overall conviction that the system is easy to use. The question that most subjects fully agreed with was that they needed little time to become familiar with the system (4.3). Furthermore, subjects, considered it as very useful and they intended to use it in the future (4.3). Moreover, all participants were of the opinion that the system helped them learn the proposed course more easily. In addition, a striking 90% of learners were pleased with the use of the system. Therefore H1 was confirmed for learners. Following the systems formative evaluation, both the system and the course were iterated. More specifically, the system s estimation on the user score was improved by developing a more intelligent code; the system database construction was also improved; and better error recovery messages, appropriately coloured, were applied. 6.2 Summative evaluation The main purpose of the summative evaluation was to assess the possible learning effects of the system according to hypotheses H4 to H7. Similar to the formative evaluation, subjects were placed into two main groups. In order to check the adaptive presentation effect, we decided to further divide each of the two main groups into another two subgroups. The students in the first subgroup had to study a course which provides adaptive presentation according to their learning style, while students of the second subgroup studied the same course with a static presentation of the content. Therefore, subjects were distinguished by the system they used and the provided or not AP. The subjects were intentionally increased to 62 and distinguished as shown in Table 2. The evaluation process was similar to the formative field s trial procedure. Subjects filled out a pre-test and a learning style questionnaire at one preliminary session which followed by another two of 2 and 3 hour-sessions for course studying. This time a detailed scenario of what they have to study was not given since this would be decrease the possible effects of adaptive navigation.

16 246 I. Kazanidis and M. Satratzemi Table 2 Groups of the experiment. The first two columns indicate the main groups and the subgroups of the experiment. The next two columns show which system used and whether adaptive presentation was applied. Numbers in the last column state the number of subjects for its sub/group Groups Subgroups System used Presentation N A ProPer 31 A1 ProPer Adaptive 15 A2 ProPer Static 16 B SCORM RTE B1 SCORM RTE Adaptive 15 B2 SCORM RTE Static Results As already stated, the main purpose of the summative valuation was to check for the possible learning effects of using ProPer. The results of the pre-test between the two main groups and the four sub groups found no significant differences on the prior knowledge of each group. Therefore, all groups were considered equal in terms of knowledge. We assumed that users can learn more using ProPer than the simple LMS used in the research (hypothesis H5). In order to confirm or reject this hypothesis, we computed a t-test for the knowledge gained (post-test pre-test score) between the students that used ProPer which provides AN (group A) and the students who used the simple LMS (group B) without AN. This time the t-test found a significant difference (p = 0.046, group A mean 25.41%, group B mean 17.96%) in the knowledge gained by the students that used ProPer for their study (Table 3), thus confirming hypothesis H5. However, a similar t-test between group 1 (A1 B1) which study with AP and group 2 (A2 B2) which show only static content found no statistically significant results. Therefore H6 was not confirmed. Table 3 Results of t-test. This table constitutes the SPSS output of the t-test that examines the knowledge gained by the two main groups of the experiment Knowledge gained Equal variances assumed Equal variances not assumed Levene s test for equality of variances F Sig. t df t-test for equality of means Sig. (2-tailed) Mean difference Std. error difference Also ANOVA was applied on the knowledge gained for each of the four subgroups of students in order to discover if students of a specific group have better performance than the rest of the students. However no significant differences were found. This could be ascribed to the small number of subjects in each group. Therefore H7 was not confirmed.

17 Improving the educational process by joining SCORM with adaptivity 247 The formative evaluation outcome about H4 was also confirmed, that is that students can complete a course faster by using ProPer, since the students in group A avoided studying unnecessary educational content and so completed the course faster than students in group B Finally participants agreed that the system is easy to use (4.3) and useful (4.4). The system thus satisfies the two main factors that conduce to the system s adoption according to TAM. 7 Conclusion This paper proposes a model for adaptive learning and presents a system called ProPer which implements the proposed model. The main objective of the system is to help learners in their education and let authors reuse educational content by adopting SCORM and effectively manage their online classes. Most AEHS do not support SCORM specifications while most SCORM compliant systems do not provide the range of adaptivity that ProPer does. Overall, as can be seen in Table 1, ProPer is the only system in comparison with the related work that provides adaptive courses, adopts SCORM specifications, includes several LMS capabilities and supports the personalisation of content presentation according to user learning style. It is this combination of all the above characteristics that is the main advantage of ProPer over related work. As far as evaluation hypotheses concerns, H1 was confirmed which show that the proposed model may lead to an attractive and effective AEHS for both learners and educators. One-to-one evaluation with educators confirmed both H2 and H3, since they stated that the adoption of SCORM is very useful especially for content reuse and ProPer may help them to better administrate their online classes by determining students and/or course weaknesses. The field trial of formative evaluation and the results of summative evaluation confirmed H4, since students that worked with ProPer were followed a better knowledge path and finished their studying earlier. However, H5 was confirmed only in summative evaluation in which students did not take any detailed scenario as in formative evaluation. This means that AN has major effects in knowledge gained especially for the students that study alone. On the other hand, it seems that AP was not so useful for students since H6 was not confirmed from the experiment. The hypothesis H6 was also not confirmed. However, a possible reason for this was the small number of participants in each subgroup. The system s evaluation of the overall results were highly satisfactory and shows that ProPer is a very useful tool for learners and educators. However, more experiments with even more participants have to be done. Up until now two courses about OOP and Java have been developed, however, our future plans are to deliver more courses complementary to traditional classroom education and undergo to more research experiments. For this reason, we have already developed an automated authoring toolkit, called Proper SAT (Kazanidis and Satratzemi, 2012), which helps authors, even without programming knowledge, to develop adaptive SCORM compliant courses.

18 248 I. Kazanidis and M. Satratzemi References ADL (2009) Sharable Content Object Reference Model. Available online at: (accessed on 23 September 2009). Brusilovsky, P. (1998) Methods and techniques of adaptive hypermedia, in Brusilovsky, P., Kobsa, A. and Vassileva, J. (Eds): Adaptive Hypertext and Hypermedia, Kluwer Academic Publishers, Dordrecht, pp Brusilovsky, P. (2003) Adaptive navigation support in educational hypermedia: the role of student knowledge level and the case for meta-adaptation, British Journal of Educational Technology, Vol. 34, No. 4, Brusilovsky, P. (2004) Adaptive educational hypermedia: from generation to generation, Proceedings of Fourth Hellenic Conference Information and Communication Technologies in Education, pp Brusilovsky, P., Karagiannidis, C. and Sampson, D. (2004) Layered evaluation of adaptive learning systems, International Journal of Continuing Engineering Education and Lifelong Learning, Vol. 14, No. 4/5, pp Brusilovsky, P., Schwarz, E. and Weber, G. (1996) ELM-ART: an intelligent tutoring system on World Wide Web, in Frasson, C., Gauthier, G. and Lesgold, A. (Eds): Intelligent Tutoring Systems, LNCS 1086, pp Carro, R., Pulido, E. and Rodriguez, P. (1999) Dynamic generation of adaptive internet-based courses, Journal of Network and Computer Applications, Vol. 22, pp Carver, C., Howard, R. and Lavelle, E. (1996) Enhancing student learning by incorporating student learning styles into adaptive hypermedia, Proceedings of ED-MEDIA 96 World Conference on Educational Multimedia and Hypermedia, AACE, Boston, MA, pp Conlan, O., Dagger, D. and Wade, V. (2002) Towards a standards-based approach to e-learning personalization using reusable learning objects, Proceedings of ELearn 2002, World Conference on E-Learning in Corporate, Government, Healthcare and Higher Education. Davis, F.D. (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly, Vol 13, No.3, pp Felder, R.M. and Silverman, L.K. (1988) Learning and teaching styles in engineering education, Engineering Education Journal, Vol. 78, No. 7, pp Gardner, H. (1993) Multiple Intelligences: The Theory in Practice, Basic Books, New York. Gütl, C. and Mödritscher, F. (2005) Towards a generic adaptive system applicable for web-based learning management environments, Proceedings of ABIS 2005, Saarbrücken, Germany. Honey, P. and Mumford, A. (1992) The Manual of Learning Styles, Peter Honey, Maidenhead, Berkshire. Honey, P. and Mumford, A. (2006) The Learning Styles Helper s Guide, Peter Honey, Maidenhead, Berkshire. Kavcic, A. (1999) Adaptation techniques in adaptive hypermedia systems, Proceedings of the 22nd International Convention MIPRO 99, Conference on Multimedia and Hypermedia Systems, Hrvaska, Croatia. Kazanidis, I. and Satratzemi, M. (2007) Adaptivity in a SCORM compliant adaptive educational hypermedia system, Proceedings of Sixth International Conference Web-based Learning (ICWL 2007), LNCS 4823, Springer, Heidelberg. Kazanidis, I. and Satratzemi, M. (2009) Adaptivity in ProPer: an adaptive SCORM compliant LMS, Journal of Distance Education Technologies, Vol. 7, No. 2, pp Kazanidis, I. and Satratzemi, M. (2012) A framework for automatic construction of reusable adaptive courses: the case of ProPer SAT 2.0, in Pena-Ayala, A. (Ed.): Smart Innovation, Systems and Technologies, Special Issue Intelligent and Adaptive Educational-Learning Systems: Achievements and Trends. Kelly, D. and Tangney, B. (2006) Adapting to intelligence profile in an adaptive educational system, Interacting with Computers, Vol. 18, pp

19 Improving the educational process by joining SCORM with adaptivity 249 Kolb, D. (1984) Experiential Learning: Experience as the Source of Learning and Development, Prentice-Hall, New Jersey. Kravcik, M. and Specht, M. (2004) Flexible navigation support in the WINDS learning environment for architecture and design, in De Bra, P. and Nejdl, W. (Eds): Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, Eindhoven, The Netherlands, August 2004, Springer, pp Merrill, M.D. (1983) Component display theory, in: Reigeluth, C.M. (Ed.): Instructional Design Theories and Models: An Overview of Their Current Status, Erlbaum Association, New Jersey, pp Morales, R. (2003) The VIBORA project, In Richards, G. (Ed.): Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, pp Papanikolaou, K.A., Grigoriadou, M., Kornilakis, H. and Magoulas, G.D. (2003) Personalizing the interaction in a Web-based educational hypermedia system: the case of INSPIRE, User- Modeling and User-Adapted Interaction, Vol. 13, No. 3, Pilar da Silva, D., Durm, R.V., Duval, E. and Olivie, H. (1998) Concepts and documents for adaptive educational hypermedia: a model and a prototype, in: Brusilovsky, P. and De Bra, P. (Eds): Proceedings of Second Adaptive Hypertext and Hypermedia Workshop at the Ninth ACM International Hypertext Conference, Pittsburgh, PA, USA. Popescu, E. (2008) Dynamic Adaptive Hypermedia Systems for e-learning, PhD Thesis, University of Craiova, Roumanie. Romero, C., Ventura, S., Hervas, C. and De Bra, P. (2005) Extending AHA!: adding levels, data mining, tests and SCORM to AHA!, Proceedings of Fifth International Conference on Human Systems Learning: Who is in control?, Morocco, pp Samarakou, M., Prentakis, P., Karolidis, D. and Papadakis, A. (2006) Formative and summative evaluation procedures for an e-learning environment: the case of e-study, GESTS International Transactions on Computer Science and Engineering, Vol. 27, No. 1, pp Stash, N., Cristea, A. and De Bra, P. (2004) Authoring of learning styles in adaptive hypermedia: problems and solutions, Proceedings of the WWW2004 Conference, pp Stern, M., Woolf, B. and Kurose, J. (1997) Intelligence on the web?, Proceedings of 8th World Conference of AIED Society, Kobe, Japan. Tessmer, M. (1993) Planning and Conducting Formative Evaluations: Improving the Quality of Education and Training, Kogan Page, London. Triantafillou, E., Pomportsis A. and Demetriadis, S. (2003) The design and the formative evaluation of an adaptive educational system based on cognitive styles, Computers & Education, Vol. 41, No. 1, pp Truong, N., Bancrof, P. and Roe, P. (2005) Learning to program through the web, ITiCSE 05, June 27 29, Monte de Caparica, Portugal. Van Marcke, K. (1992) A generic task model for instruction, in: Dijkstra, S., Krammer, H.P.M. and van Merrienboer, J.J.G. (Eds): Instructional Models for Computer-Based Learning Environments, NATO ASI Series F, Vol 104, Springer-Verlag, Berlin, Heidelberg, pp Witkin, H.A., Moore, C.A., Goodenough, D.R. and Cox, P.W. (1977) Field-dependent and field independent cognitive styles and their educational implications, Review of Educational Research, Vol. 47, No. 1, 1 64.