Red de Revstas Centífcas de Amérca Latna, el Carbe, España y Portugal Sstema de Informacón Centífca Eduardo Islas, Mguel Pérez, Gullermo Rodrguez, Israel Paredes, Ivonne Ávla, Mguel Mendoza E-learnng Tools Evaluaton and Roadmap Development for an Electrcal Utlty, vol. 2, núm., aprl, 2007, pp. 63-75, Unversdad de Talca Chle Avalable n: http://www.redalyc.org/artculo.oa?d=9652006 Journal of Theoretcal and Appled Electronc Commerce Research, ISSN (Electronc Verson): 078-876 ncerpa@utalca.cl Unversdad de Talca Chle How to cte Complete ssue More nformaton about ths artcle Journal's homepage www.redalyc.org Non-Proft Academc Project, developed under the Open Acces Intatve
63 E-learnng Tools Evaluaton and Roadmap Development for an Electrcal Utlty Eduardo Islas, Mguel Pérez, Gullermo Rodrguez, Israel Paredes 2, Ivonne Ávla and Mguel Mendoza Insttuto de Investgacones Eléctrcas, Cuernavaca, Morelos, Méxco, eslas@e.org.mx, mperez@e.org.mx, gro@e.org.mx, von_ag@yahoo.com.mx, mmendoza@e.org.mx 2 Comsón Federal de Electrcdad (CFE), Méxco DF., Méxco, paredes@cfe.gob.mx Receved 7 August 2006; receved n revsed form 5 January 2007; accepted 05 March 2007 Abstract Ths paper descrbes a methodology based on 40 crterons as well as a mxture of MCDM methods to evaluate dfferent technologes appled n modern tranng systems. The crterons are grouped n a threedmensonal (3D) model n accordance wth ther use and applcaton n tranng processes. The proposed 3D model ncludes the Management (M), the Technologcal (T) and the Instructonal (I) dmensons. Applyng ths methodology we can evaluate dfferent tranng and learnng technologes. Furthermore a technology roadmap s outlned usng the obtaned results n the evaluaton. Ths roadmap wll provde CFE (Comsón Federal de Electrcdad whch s the Natonal Electrcty Utlty n Mexco) wth elements to decde, enhance and upgrade ts current tranng nformaton systems or acqure new ones as well as showng what courses are worth mplementng wth e-learnng technologes and when these courses should be programmed. Key words: Roadmap, e-learnng, learnng objects, evaluaton, MCDM, methodology
64 Introducton The 3D model and the proposed methodology n ths paper, not only are helpful to evaluate the applcablty of each learnng tool from a global pont of vew, but they are also useful to establsh the utlzaton of each learnng tool n every dmenson (M, T, I), n every plane (MT, MI, TI) and n a 3-dmensonal space (MTI). Ths provdes dfferent vews of the same tool, whch allows evaluatng each tool from dfferent perspectves. These perspectves help to determne whether or not a tool fulflls the requrements from an M-T-I pont of vew. The management dmenson evaluates aspects such as: Course Management, Student Trackng, Currculum Management, etc. The nstructonal dmenson ncludes Instructonal Desgn, Ddactc Plannng, Content Producton, Instructor Manual, Student Manual, and so on. Fnally, the technologcal dmenson nvolves the current software and hardware tools avalable as canddates for acquston and use n the tranng processes of a company. These 3 conceptual dmensons outlne a 3D space and three planes (see Fgure ). All of these represent dfferent relatonshps among them. The elements of the management-technologcal plane or quadrant relate how technology s beng appled n course admnstraton and how much the cost of hardware and software s. The aspects of the management-nstructonal quadrant relate how we can admnstrate the content of courses and keep track of student s progress and performance of hs/her tranng actvtes. The ssues n the technologcal-nstructonal plane relate how software and hardware technology s used to develop content for courses. And fnally the managementtechnologcal-nstructonal space ndcates how the technology s beng used to manage the learnng process. Instructonal MI plane TI plane MTI space Management MT plane Technologcal Fgure : Three-dmensonal model proposed Commercally avalable tools ntegrate dfferent modules provdng a complete learnng tool. However, ths ntegraton ncreases the cost and complexty of each tool. In ths paper we present evaluaton results for some Learnng Management Systems (LMSs), Learnng Content Management Systems (LCMSs) and Content Management Systems (CMSs). The CMSs are complementary tools to obtan a more complete learnng tool. The proposed methodology was used to evaluate three commercal platforms (Blackboard, IBM Lotus and PeopleSoft) and two n-house developed tools (SIC, and UTEC). The SIC and UTEC tools are systems used by CFE to manage ts Human Captal to algn workforce wth busness objectves. The rest of the paper s organzed as follows: secton 2 summarzes a revew of evaluaton methodologes; secton 3 ncludes some e-learnng concepts related to the tools evaluated n ths paper; secton 4 descrbes the evaluaton methodology used; secton 5 llustrates how the methodology s appled to evaluate e-learnng tools, also shows the evaluaton outcomes and ncludes an e-learnng roadmap slde, whch s based on the evaluaton outcomes; secton 6 ncludes some conclusons and fnally secton 7 provdes a lst of references. 2 Related Work Related wth ths work n [5] some course management systems are compared and evaluated takng nto account the same crterons appled n ths work, but wthout the assgnment of several characterstcs to each crteron and wthout gvng dfferent weghts to each crteron based on ther relevance as we are proposng n ths work. Addtonally n [5] the authors are not consderng the 3-dmensonal model that we are usng n order to evaluate the tools from dfferent perspectves such as: nstructonal, management and technologcal. To evaluate dfferent alternatves and make a decson, several methods have been proposed n the lterature; the methods can be appled n dfferent assessments: Multobjectve [4], Multcrtera [9] or Multattrbutes [20].
One of the most used assessments s the MultCrtera Decson Makng (MCDM), n whch the methods can be dvded n three categores [2], []: Weghtng methods: These methods assgn dfferent weghts to each crteron dependng on ther degree of mportance wth respect to the other crterons, some of these methods are: pont allocaton, herarchcal pont allocaton, swng weghtng, tradeoff weghtng, revson of weghts, nonherarchcal weght assessment and herarchcal weght assessment. Determnstc rankng methods: The optons are ranked based on the evaluaton carred out usng prevously obtaned values, some of these methods are: addtve lnear value functon, multplcatve value functon, non lnear value functon, goal programmng and ELECTRE. Uncertanty rankng methods: The evaluaton takes nto account smulated values to rank the choces and make a decson, some of these methods are: lnear utlty functon, non lnear utlty functon, regret and stochastc domnance. In ths work we used the combnaton of three methods: herarchcal weght assessment, addtve lnear value functon and goal programmng n order to compare the results obtaned wth the applcaton of dfferent methods and based on our results compare the weghtng methods and amalgamaton rules n terms of ther ease of use, approprateness and valdty. The methodology used n ths work s based on some concepts from the methodology developed n [7], whch we appled n the evaluaton of software tools to develop vrtual realty systems. In ths work we are evolvng ths methodology addng the evaluaton of the alternatves usng other two MCDM methods and fnally we are usng a proposed 3D model and other crterons to evaluate the software tools related wth learnng and content management systems. It s worth pontng out here that n some degree the MCDM are general purpose methods, n the sense that dependng on the knd of tems to be evaluated, they demand the defnton of a set of crtera (or parameters) that an tem n turn must accomplsh wth. The set of crtera n turn personalses the methodology. However the sets of steps nvolved n a methodology mght reman unaltered. That s, the crtera are dfferent but the methodology s the same. Thus, usng a MCDM methodology, we can evaluate dfferent knds of tems such as LMSs or VR tools, dfferent types of hardware, etc., as long as we establsh an approprate set of crtera n each case. 3 Concepts 3. E-learnng E-learnng conssts of provdng educatonal programs and learnng systems by means of electronc devces, such as: computers, CDs, nternet or ntranet, multmeda among others technologes. The electronc educaton or e- learnng nvolves a group of applcatons and tools, such as: web based learnng, computer based tranng, vrtual classrooms and dgtal collaboraton (group work). These tools have many advantages: hgher productvty, smultaneous new product processes and delvery of applcatons to many partcpants. Partcpants can make ther own learnng path and perhaps the best e-learnng contrbuton, whch orgnates that the total cost per partcpant s lower than the expenses n a tradtonal nstructor guded system. Nevertheless, desgn and development of e- learnng programs can ntally be hgher. Among e-learnng hardware and software tools we can menton the followng tools: web servers, CMS, LMS, LCMS, collaboraton tools and vdeo servces [4], [5]. The followng sectons descrbe some of these e-learnng tools, manly those related to learnng and content management systems. Content Management Systems. A Content Management System (CMS) s a combnaton of a large database, fle systems and other related software modules whch are used to store and later retreve huge amounts of data. These systems are dfferent from the databases n the sense that these can ndex and use text, audo clps, vdeo clps, or mages. Users of the content management system can fnd relevant content wthn a database searchng by keywords, authors, date of creaton, etc. Content Management Systems can also be used to create nformaton portals whch serve as the backbone of data management. They are usually based on a pre-wrtten template that acts as a platform for each page n the ste whle those pages are beng created [7]. Learnng Management Systems. An LMS s a hgh-level, strategc soluton for plannng, delverng, and managng all learnng events wthn an organzaton, ncludng vrtual classrooms and nstructor-guded courses. The prmary soluton conssts of replacng solated and fragmented learnng programs wth systematc means of assessng and rasng competency and performance levels throughout the organzaton. Fgure 2 shows the schematc operaton of an LMS nsde of a company. For example, an LMS smplfes certfcaton efforts, enables companes to algn learnng ntatves wth strategc goals, and provdes vable means of enterprse-level sklls management. The focus of an LMS s to manage learners, keepng track of ther progress and performance across all types of tranng actvtes. It performs heavy-duty 65
66 admnstratve tasks, such as reportng to human resources department and other systems but t s not generally used to create course content [8], [3], [4]. Fgure 2: Schematc Operaton of an LMS [2] Learnng Content Management Systems. An LCMS s a mult-user envronment where learnng developers can create, store, reuse, manage, and delver dgtal learnng content from a central object repostory, see Fgure 3. LCMSs motvate personalzed learnng for students and help corporatons n reducng the dstance between tool development and the LMSs. If corporatons and educatve nsttutons wsh to take advantage of ths technology, they wll frst have to assess ther needs and ncorporate e-learnng solutons wthn a corporatve strategy [3], [3], [4]. 4 Evaluaton Methodology Fgure 3: Schematc Operaton of an LCMS [2] In the next four subsectons the complete proposed methodology s descrbed n several steps from crterons selecton (a) to show the results obtaned and conclusons (g). 4. Three-dmensonal model The model proposed n ths work s used to analyze modern tranng systems. Ths model relates the three more mportant aspects nvolved n personnel tranng: Management, Technologcal and Instructonal. Dfferent analyss, evaluatons or studes can be made wth the 3D model proposed; therefore the evaluaton of a specfc technologcal capacty (a pont n the technologcal dmenson) wll be a pont n a 3D coordnate system. Every pont n the model (see Fgure) represents the use, applcaton or evaluaton of the technologcal capacty wth respect to the other two dmensons: nstructonal and admnstratve. The use of the 3D model s a common practce n CFE tranng actvtes and t was a requrement to algn the crtera to ths model that CFE uses to rule tranng.
67 4.2 Crterons and weght defnton The methodology s based on 40 crterons used to evaluate dfferent technologes appled n modern tranng systems. Ths evaluaton methodology, personalzed wth an approprate set of crtera, has been appled earler n the evaluaton of software and hardware tools, whch are related wth development of vrtual realty systems [6], [7]. The crterons are grouped n the 3D model descrbed above n accordance wth ther use and applcaton n the tranng processes. a) Crterons selecton. In ths step a group of crterons are defned. Each crteron evaluates the degree of accomplshment of a user s requrement for hs/her tranng process. These crterons mght even be useful as a gude to develop learnng and content management systems. b) Value assgnment. Ths s acheved by assgnng to every crteron a scale from 0 to 5 to dentfy dfferent level of accomplshment of the feature assocated wth that crteron. c) Weght assgnment. Dfferent weghts usng non-herarchcal and herarchcal weght assessment are assgned to each crteron such that the most mportant crterons for the fnal users are emphaszed. The values assgned for these weghts are n a scale between and 2. Most mportant crterons are assgned 2 whle s assgned to less mportant ones. The purpose of the herarches s to make weghtng easer; n theory, how the herarchy s structured should no affect the fnal weght assgned for each crteron [] and n experments carred out n [6] and [20] non-herarchcal weghts tend to be flatter (more equal), whle herarchcal weghts are steeper (have a greater varance). d) Tools selecton. A selecton of software tools must be carred out based on the relevance of the company or nsttuton whch has developed and supported the software package. Addtonally the relevance of the software tool was consdered n accordance wth the number of users and recommendatons found n most LMS and LCMS references. 4.3 Evaluaton methods defnton The use of three amalgamaton methods usng the combnaton of MCDMs s to compare the weghtng methods and value functons n terms of ther ease of use, approprateness and valdty [2], []. e) Analyss and evaluaton of each software tool. Based on the crterons, the weght assgnment and three combnaton of methods to evaluate dfferent optons, the software tools were revewed, analyzed and evaluated, gradng them n accordance wth each method appled: MCDM. Addtve value functon and non-herarchcal weght assessment MAX V n ( x j ) = wv ( xj ) = () where: x = The value of crteron for alternatve j j v ( x j ) = A sngle crteron value functon that converts the crteron nto a measure of value or worth. These are often scaled from 0 to, wth more beng better. In ths frst method these values were not scaled w = Weght for crteron, representng ts relatve mportance. These are often normalzed so w = In ths frst method all the weghts were assgned wth the same value of n = Number of crterons MCDM 2. Addtve value functon and herarchcal weght assessment MAX V n ( x j ) = wv ( xj ) = (2) where:
68 v ( x j ) = In ths second method these values were scaled from 0 to usng the followng expresson: 5 xj v ( xj ) = + 5 w = In ths second method the herarchcal weght assessment was used The MAX n () and (2) ndcates that hgher values are better MCDM 3. Goal programmng and herarchcal weght assessment Goal programmng focuses on achevement of goals, as oppose to addtve value functons, whch emphasze tradng off crtera. The results obtaned at the end of the paper shown the same rankng of choces t does not matter the method used as opposed n [] MIN V where: [ j ] n ( x ) = w g v ( x ) j = p v ( x j ) = Also n ths thrd method these values were scaled from 0 to (3) w = Also n ths thrd method the herarchcal weght assessment was used g = The goal for crtera, defned as an acceptable, desrable or deal. In goal programmng, v ( x j ) are usually lnear functons of the x j p = Exponent appled to the absolute value of the weghted dfference between the goal and the actual value. In ths thrd method was used p=, whch s often called cty block metrc MIN n (3) ndcates that smaller values are better 4.4 Results f) Comparson of software tools. In ths step the results obtaned are represented n a graph, comparng them and determnng ther poston and mportance level wth respect to other tools (benchmarkng graphs). g) Results obtaned and conclusons. Fnally, for each software tool the advantages and dsadvantages are descrbed and thus the compettve poston wth respect to the other software tools, to create learnng and content management systems, s determned. 5 Applcaton of the Evaluaton Methodology 5. Three-dmensonal model The 3D model shown n Fgure 4 s outlned for three axes or dmensons: the Management dmenson (M) related wth the admnstraton tasks of the e-learnng system, the Technologcal dmenson (T) assocated wth the use of software and hardware resources and fnally the Instructonal dmenson (I) that conssts of the development of the courses contents. Usng the proposed methodology, dfferent Learnng Management Systems (LMS), Content Management Systems (CMS), Learnng Content Management Systems (LCMS) and other technologes for enterprse tranng and learnng can be analyzed and evaluated. 5.2 Crterons defnton and value assgnment In Fgure 4, all defned crterons are shown. These crterons were taken from [5] and these were grouped for each dmenson (the crterons that nvolve only one dmenson), n each plane (the crterons that nvolve two dmensons) or n the space (the crterons that nvolve three dmensons). All the crterons were analyzed, modfed and grouped n collaboraton wth the people responsble of the tranng processes at CFE.
69 2. Instructonal Dmenson (I) 4. MI plane 4. Course Management 5. TI plane 5. Instructor Helpdesk 5.2 Course templates 5.3 Instructonal standards complance 5.4 Searchng wthn course 5.5 Onlne gradng tools 5.6 Dscusson Forums 5.7 Bookmarks 5.8 Self- assessment 3. Technologcal Dmenson (T) 3. Clent Browser Requred 3.2 Server Software 3.3 Database Requrements 3.4 Open Source 3.5 Software Verson 3.6 Accessblty complance 6. MT plane 6. Calendar/Progress Revew 6.2 Authentcaton 6.3 Hosted servces 6.4 Regstraton Integraton 6.5 Fle Exchange 6.6 Internal Emal 6.7 Onlne journal/notes 6.8 Real-tme Chat 7. MTI space 7. Content sharng/reuse 7.2 Automated testng and scorng 7.3 Instructonal Desgn Tools 7.4 Customzed look and feel 7.5 Course Authorzaton 7.6 Optonal Extras. Management Dmenson (M). Student Trackng.2 Currculum management.3 Orentaton/Help 6.9 Vdeo Servces 6.0 Whteboard 6. Group work 6.2 Student communty buldng 6.3 Student Portfolos 6.4 Work offlne/synchronze 6.5 Lcense Costs 6.6 Company profle Fgure 4: Three-dmensonal model to evaluate modern tranng systems Although subjectve, t s worth clarfyng that ths groupng s based on experence and needs of CFE s decson maker and could be grouped n dfferent ways, for nstance some crterons can be ncluded n a plane nstead of n a dmenson. For nstance student trackng was classfed as M (because student trackng usually s a management actvty) but t could have been classfed n the MT plane (because ths knd of management s acheved usng LMS technology), dependng on evaluator nterest on dfferent aspects that a crteron mght nvolve. Nevertheless, the methodology s accuracy remans unaltered snce each crteron s evaluated ndvdually and ts value s the same no matter where t s grouped. In ths paper, due to lack of space only one crteron s descrbed n detal, as well as ts value assgnment. The rest of the crterons were analyzed n the same way and are descrbed n detal n [8]. Student Trackng (Management dmenson). Student trackng s the system ablty to track the usage of course materals by students, and to perform addtonal analyss and reportng,.e. the usage by each student and by the whole group. Student Trackng tools nclude statstcal analyss of student performance data and progress reports for ndvdual students n the course. Progress reports generally consst of both actvtes and the tme stamps when an actvty occurred [5]. Features. Only the trackng of exams can be carred out. 2. Every element can be selected to carry out the student s trackng (homework, tests, essays, fnal exam, projects, etc.) 3. Emssons of reports of every element n the course. 4. Dfferent reports can be selected. 5. The reports can be confgured to present one or several elements. Scale Descrpton 0 Student trackng s not supported. The tool has one of the features above 2 The tool has two of the features above 3 The tool has three of the features above 4 The tool has four of the features above 5 The tool has at least fve of the features above Table : Value assgnment for Student Trackng Addtonal to the evaluaton of the systems usng the 40 crterons, other actvtes nvolved n the analyss of modern tranng systems can be related to the 3D model, for example:
70. The management-technologcal plane or quadrant may nclude the cost-beneft analyss obtaned f the company whshes to purchase or develop the e-learnng technologes analyzed. 2. The aspects of the management-nstructonal quadrant nvolve also the tranng needs that correlate the company labor competences and the soluton of specfc problems that the enterprse faces. 3. Other ssues relatng the technologcal-nstructonal plane are, for example, practcal gudelnes to optmze the use of technologes for nstructonal purposes and to reach the goal that an e-learnng course could be dependng on context, as effectve as or better than a tradtonal one. Regardng completeness of the crtera used n ths methodology, we only took the set reported n the lterature [5], because t fulflled and exceeded CFE s needs. We were not engaged n accomplshng a general completeness of a set of crtera. 5.3 Results The followng subsectons show the results obtaned for the commercal systems: Blackboard, IBM Lotus and PeopleSoft and the CFE s systems SIC and UTEC. In the frst subsecton the results obtaned for each dmenson and plane are shown n order to realze the accomplshment of each system from dfferent perceptons: Management, Technologcal and Instructonal perspectves. The second subsecton shows the results obtaned for each system applyng the three amalgamaton methods descrbed above, and the thrd subsecton shows how we appled the results to obtan a roadmap, whch wll help CFE to make decsons not only to choose the best LMS tools for tranng processes, but also to decde whch are the hardware and software tools that CFE should use for course development and management. 5.3. Results obtaned for e-learnng systems from dfferent perspectves The evaluaton was carred out by assessng the degree of fulfllment of features of each crteron. Each e-learnng tool was tested and evaluated developng a tranng course prototype for valves mantenance [] and n collaboraton wth software provders each feature of each crteron was revewed. Due to lack of space, only the results for the technologcal-nstructonal plane are presented n a graphcal way (see Fgure 5) and the rest of groups are presented n a tabular form n Table 2. It s very mportant to say that not all systems evaluated belong to the same type; however the methodology was appled under customer requrement n order to know how each system fulflls content and learnng management n tranng processes. Instructor helpdesk Course templates Instructonal standards complance Searchng wthn course Onlne gradng tools Dscusson forums Bookmarks 5 4 3 2 5 4 3 2 Self-assessment 0 IBM 0 People Soft 5 5 5 4 4 4 3 3 3 2 2 2 0 0 SIC UTEC Blackboard 0 Fgure 5: Graphcs obtaned for the technologcal-nstructonal plane
7 Group System Evaluaton Group System Evaluaton Group System Evaluaton Management IBM 3 out of 5 Technologcal- IBM 32 out of 40 Management- IBM 26 out of 30 Dmenson PeopleSoft 5 out of 5 Instructonal PeopleSoft 24 out of 40 Technologcal- PeopleSoft 3 out of 30 SIC 0 out of 5 Plane SIC 2 out of 40 Instructonal SIC 0 out of 30 UTEC 6 out of 5 UTEC 3 out of 40 Space UTEC 0 out of 30 Blackboard out of 5 Blackboard 32 out of 40 Blackboard 25 out of 30 Group System Evaluaton Group System Evaluaton Group System Evaluaton Technologcal IBM 7 out of 30 Management- IBM 5 out of 5 Management- IBM 53 out of 80 Dmenson PeopleSoft 23 out of 30 Instructonal Plane PeopleSoft 5 out of 5 Technologcal- Instructonal PeopleSoft 55 out of 80 SIC 8 out of 30 SIC 0 out of 5 Space SIC 2 out of 80 UTEC 0 out of 30 UTEC out of 5 UTEC 26 out of 80 Blackboard 5 out of 30 Blackboard 4 out of 5 Blackboard 62 out of 80 Table 2: Results obtaned for all groups n the 3D model 5.3.2 Results obtaned applyng three amalgamaton MCDM methods The results for the frst MCDM method are depcted n Fgure 6, whch shows the rankng and global results for each software tool. These global results nclude all the crterons consdered applyng the addtve value functon wthout v and usng non-herarchcal weght assessment. scalng the value functon ( ) x j It s mportant to note that PeopleSoft and IBM Lotus are consdered to be LMSs whle Blackboard s a tool only to manage content (CMS). SIC s a computatonal system to regster and mantan nformaton about employee tranng n CFE and UTEC s a web based learnng system to teach personnel about dfferent subjects related to actvtes n CFE. These last two systems were not desgned to manage content and learnng systems however, they have some LMS s features. 200 75 50 49 46 35 25 00 75 66 50 32 25 0 a) Commercal CMSs Blackboard b) Commercal LMSs IBM Lotus People Soft c) CFE s systems SIC UTEC Fgure 6: Total results obtaned applyng the frst MCDM method for the systems evaluated In Fgure 7 are shown the results obtaned for the second MCDM method applyng the addtve value functon wth the scalng of the value functons from 0 to and usng herarchcal weght assessment. In the frst and second method greater values mean better electons.
72 75 50.4 49.7 50 43.9 25 2.8 0 0 Fgure 7: Total results obtaned applyng the second MCDM method for the systems evaluated Fnally n Fgure 8 the results obtaned for the thrd MCDM method are shown, n ths thrd amalgamaton method we appled goal programmng and herarchcal weght assessment, n ths method the smaller values mean whch optons are better. 75 5.9 50 40 25 4.5 5.5 22.4 0 Fgure 8: Total results obtaned applyng the thrd MCDM method for the systems evaluated
73 Based on the statements made n [] and the results obtaned n ths work we can compare the weghtng methods and amalgamaton rules n terms of ther ease of use, approprateness and valdty. Ease of Use and Approprateness. Based on [] the ratng weghts are easer to understand and use than tradeoff weghtng, and that ratng does a better job of helpng users to understand the problem; therefore we used ratng weghts n ths work, wth ratng weghts we confrm the ease of use of ths method because we dd not have to do addtonal calculatons to found the tradeoffs between crterons weghts. In the experments and questonnares made n [] the majorty of the users felt that goal programmng method makes sense and s logcally sound; mproves consstency of judgment; and ads understandng of the problem n comparson wth applyng other MCDM methods or holstc assessment (drect evaluaton based on users expertse). The questonnares also asked whether goal programmng, addtve value functons, or holstc judgment was preferred. The users sad to prefer usng goal programmng because the goal settng exercse gve addtonal nsght. Wth the use of amalgamaton methods usng goal programmng we corroborate n ths paper that goal programmng mproves consstency of judgment and helps to better understand the problem, because we have to know a pror f we want a modest or ambtous goal. Convergent Valdty: Do Dfferent Methods Yeld the Same Optons Ranks? Hobbs and Meer n [] answer two questons related to convergent valdty. The frst queston s, how much do people agree on the ranks of the optons? Ther answer s: when a MCDM method s used, there s some but not complete agreement among the partcpants as to whch optons are best, and the use of goals results n greater dsagreement. Further, MCDM methods result n more agreement among partcpants that does holstc judgment because MCDM methods do not have prejudces and consder all crtera at the same tme, therefore such methods should yeld greater convergence of opnon, In ths paper the evaluaton was carred out by several people applyng the same methods, the same crterons and same weghts n order to avod dsagreement n the results. The second queston was: do dfferent methods yeld dfferent opton ranks? The answer was that dfferences n opton ranks between pars of methods appear less mportant than the nterpersonal dfferences and the choce of an MCDM method affect ranks less than who chooses the method, t agrees wth [0] but dsagrees wth [9]. The results obtaned n ths work agrees wth [] because the results show hgh correlaton between them, n fact the ranks obtaned n the three amalgamaton methods are the same t does not matter whch weghtng method and whch determnstc method we appled (see fgures 6, 7 and 8) 5.3.3 Roadmap The core of our contrbuton s n the detecton and classfcaton of a set of concepts whch are used to evaluate e- learnng systems and to dentfy the capabltes, propertes or characterstcs that structurally descrbe types of requrements and defne the key elements to be ncluded n an e-learnng envronment. These elements or crtera and the methodology are descrbed n detal n [7], whch are the bass to buld a technologcal roadmap that s the gude for the CFE s management personnel to make decsons not only about whch are the best LMS tools for ts tranng processes, but also whch are the hardware and software tools that CFE should use for courses development and ther management. There are many proposals for modern tranng systems requrements; each one has ts own elements. Some use the same concepts but the names are dfferent, whch makes t complex and laborous to compare. The approach presented here unfes the varous termnologes, ncreases the knowledge about e-learnng systems whch helps to select the tool that s more approprate to CFE s needs ether f the tool s gong to be n home developed or acqured from a provder. The roadmap proposed suggests to CFE what projects should develop n the next sx years, based on the drvng factors, the nternal needs detected n CFE, and n the avalable technologes n the next years (see Fgure 9). There are many aspects whch an LMS mght nfluence, for nstance there s a cultural mpact regardng tranng management, a computer s gong to partcpate n teachng and management, avalablty of courses and tutors, etc. Ths s why CFE needs not only to buy and use an LMS, but t also has to work n all these aspects. For the tme beng and because CFE already had an LMS, whch by the way, does not have the maxmum score n the evaluaton, t was recommended to use the one they already have to start the work. However and due to the cost of the LMS on one hand and the portals development experence of CFE on the other, t was also suggested to start an n house LMS development. 6 Conclusons and future work In the applcaton of MCDM methods to make a decson based on the results, we conclude as n [] that t s recommendable to apply more than one approach because dfferent methods offer dfferent results to compare, n ths case, goal programmng and addtve value functons are suggested and besdes the results must be shown to decson makers who can mull over the dfferences or confrm the resemblances. In evaluatng the results of dfferent methods, the potental for bases should be kept n mnd. The extra effort s not large; the potental benefts, n terms of enhanced confdence and a more relable evaluaton process, are worth.
[] A. Ayala, M. Pérez and G. Rodríguez, Descrpcón del prototpo de un curso de mantenmento a válvulas, Insttuto de Investgacones Eléctrcas, Cuernavaca Méxco, Techncal Report 2692/GSI/D008, 2005. [2] M. L. Bell, B. F. Hobbs, E. M. Ellott, H. Ells and Z. Robnson, An Evaluaton of Multcrtera Decson Makng Methods n Integrated Assesment of Clmate Polcy, n Lecture Notes n Economcs and Mathematcal Systems. Research and Practce n Multple Crtera Decson Makng, Vrgna USA, 998, pp 229-237 [3] M. Brennan, S. Funke and C. Anderson (200, May), The Learnng Content Management System, [Onlne]. Avalable: http://www.nternettme.com/learnng/lcms/idclcmswhtepaper.pdf 74 Tranng Processes Roadmap at CFE (2005-20) 20) 2 years 4 years 6 years Drvng Factors To enhance and mprove learnng and tranng actvtes for the Insttuton s human captal Lower prces related wth the use of software and hardware tools Internal needs at CFE To mantan and ncrease tranng ndcators, enhancng at the same tme the human captal value To optmze the expenses for learnng and tranng actvtes To ntegrate solated systems for management and development of the human captal at CFE (SIC, SICEC, UTEC, IBM Lotus) Avalable Technologes Commercal Learnng Management Technologes Internet Hgh speed networks Technology alternatves to develop content: Multmeda (nteractve and dynamc anmatons (Flash), dynamc webstes (Dream weaver), vrtual realty tools (OpenSceneGraph), dynamc presentatons (Power Pont), among others). Research and Development Projects Development of tranng courses avalable through nternet to support and ncrease the human captal value Use of a commercal LMS (IBM Lotus) Vrtual realty devces (Gloves, HMDs, Trackers, etc.) Development of mmersve tranng courses Development of an own LMS Use of the LMS developed Fgure 9: Tranng Processes Roadmap at CFE The proposed model s beng appled at CFE, a 70,000 employee company that generates, transmts, and dstrbutes electrcty through out the naton. One of ts strategc objectves s to establsh tranng programs to certfy ts employees n labor competences algned wth ts msson and future vson. Although the model can be used to analyze a broad varety of dfferent e-learnng technologes, the paper only address asynchronous web-based envronments where learnng content or courseware s served from a web server and delvered on demand to the learner s workstaton. Learners can thus make progress by themselves. The courseware may be comprsed of any combnaton of text, mages, anmaton, sounds and moves. The courseware s nteractve and s often combned wth some type of assessment. One of the man benefts for CFE after ths evaluaton s that personnel n charge of CFE s tranng management are now well nformed about ths technology. Furthermore, they have now a roadmap for modern tranng systems technology. Thus, they can make up an acton plan and choose the best paths to follow, n order to ntegrate ths technology nto CFE s processes. For nstance, the roadmap recommends begnnng mmedately the use of the tool already avalable n CFE (Lotus s LMS), whch s not the system wth the best score n the evaluaton, but t s stll acceptable. Ths would avod spendng more resources on new lcenses. It s also recommended to develop smultaneously ther own LMS whch suts CFE s needs. Ths wll avod spendng resources on annual lcense fees and they wll have the source code of ther own LMS. Ths wll enable CFE to make changes and mprovements under demand. Ths mght be acheved n a perod of 3 to 5 years. References
75 [4] V. Chankong and Y. Hames, Multobjectve Decsón Makng: Theory and Methods, North Holland, Amsterdam, 983. [5] Edutools (2007, Jan.) CMS Home, Edutools, [Onlne]. Avalable: http://www.edutools.nfo/course/. [6] T. Eppel, Descrpton and Procedure n Varance n Multattrbute Utlty Measurement, School of Management, Purdue Unversty, West Lafayette, 992. [7] ERPToday (2006) Who needs a Content Management System (CMS)?, Knowledgehlls, Avalable: http://erptoday.knowledgehlls.com/cms/who-needs-content-management.aspx [8] L. Greenberg (2002, Dec.) LMS and LCMS: What's the Dfference?, ASTD, Avalable: http://www.learnngcrcuts.org/2002/dec2002/greenberg.htm [9] B. F. Hobbs, What Can We Learn From Experments n Multobjectve Decson Analyss?, IEEE Transactons Systems, Man Cyber., Vol. SMC-6, pp. 384-396, May 986. [0] B. F. Hobbs, V. Chankong, W. Hamadeh and E.Z. Stakhv, Does Choce of Multcrtera Method Matter? An experment n Water Resources Plannng, Water Resources Res., Vol. 28 (7), pp. 767-779, July 992. [] B. F. Hobbs and P. M. Meer, Multcrtera Methods for Resource Plannng: An expermental comparson, IEEE Transactons on Power Systems, Vol. 9 (4), pp. 8-87, November 994. [2] W. Horton (2007, Jan.) E-learnng tools and technologes, Wllam Horton Consultng. Inc., Avalable: http://www.horton.com. [3] W. Horton, Desgnng Web.Based Tranng, John Wley & Sons, Inc; USA, 2000. [4] W. Horton and K. Norton, E-learnng tools and technologes, Wley Publshng, Inc; Indanapols USA, 2003. [5] Informátca Mlenum (2005, Jan.) E-Learnng: el futuro de la educacón a dstanca, Informátca Mlenum, S.A. de C.V., Avalable: http://www.nformatcamlenum.com.mx/pagnas/mn/artculo78.htm [6] E. Islas, E. Zabre and M. Pérez, Evaluacón de herramentas de hardware y software para el desarrollo de aplcacones de realdad vrtual; Boletín IIE, vol. 28, pp. 6-67, Apr-Jun 2004. [7] M. Pérez, E. Zabre and E. Islas, Prospectva y ruta tecnológca para el uso de la tecnología de realdad vrtual en los procesos de la CFE, Insttuto de Investgacones Eléctrcas, Cuernavaca Méxco, Techncal Report IIE/GSI/022/2003, 2003. [8] G. Rodríguez, E. Islas, M. Pérez, I. Ávla and M. Mendoza, Metodología para la evaluacón de tecnologías para los sstemas modernos de capactacón, Insttuto de Investgacones Eléctrcas, Cuernavaca, Méxco, Techncal Report 2692/GSI/D00, 2004. [9] T.J. Stewart, A Crtcal Survey on the Status of Multple Crtera Decsón Makng Theory and Practce, OMEGA The Internatonal Journal of Management Scence, vol. 20 (5/6), pp. 569-586, 992. [20] W.G. Stllwell, D. von Wnterfeldt and R. S. Jhon, Comparng Herarchcal and Nonherarchcal Weghtng Methods for Elctng Multattrbute Value Models, Manage Scence, vol. 33 (4), pp. 442-450, Aprl 987.