Iformatio System for the valuatio of Uiversities i Spai M. Socorro Garcia-Cascales 1, M. Teresa Lamata 2 1 Dpto de Electróica, Tecología de Computadoras y Proyectos. Uiversidad de Politécica de Cartagea. Murcia, España 2 Dpto. Ciecias de la Computació e Iteligecia Artificial. Uiversidad de Graada. 18071- Graada. España Email: socorro.garcia@upct.es, mtl@decsai.ugr.es Abstract The Spaish Natioal Agecy for Quality Assessmet ad Accreditatio as part of its evaluatio activities has established a procedure for evaluatig both teachig ad istitutios, by meas of the Istitutioal Assessmet Programme. I this commuicatio we shall focus o the exteral assessmet phase for qualificatios i the field of Idustrial Egieerig ad specifically o the structures of the database for a Decisio Support System o the uiversities rakigs. I particular, this paper will focus o obtaiig the weight of the criteria ad defiitio of the liguistic labels used i the exteral assessmet phase. Keywords Aalytical Hierarchy Process (AHP), Liguistic Labels, Qualificatio evaluatio, Weightig Criteria facilitate the process of evaluatio o the uiversities rakigs. Aother aim is to obtai the fuzzy umbers associated with the liguistic labels used to evaluate the ANECA i the exteral assessmet phase. The paper is orgaised as follows: The ext sectio itroduces the ANECA process. Sectio 3 itroduces the liguistic variables ad fuzzy sets. I sectio 4, the framework for the AHP method is defied. Sectio 5 describes the database. Sectio 6 shows the aggregatio results ad fially we outlie the most importat coclusios. 1 Itroductio The uiversity rakigs which just 20 years ago were a iovatio are today a ormal characteristic i the majority of coutries with extesive systems of higher educatio [7, 13, 16, 17, 18]. These lists have a icreasig impact ot oly betwee the uiversities themselves, but also betwee differet social sectors. Uiversity rakigs are lists of certai groups of istitutios, classified i comparative form, ad accordig to a commo set of idicators, i descedig order. Likewise, the reorgaisatio at Europea scale of uiversity studies as a result of the Bologa Process, will cotribute i a active way to the harmoisatio of basic Europea academic aspects. This aspect will allow, o the oe had, for a better coectio betwee uiversities ad o aother had, for a easier compariso betwee them; for which certai idicators of fuctioig will be ecessary. All the above leads us to thik about the eed for the existece of global rakigs of uiversities as a istrumet to measure their quality rigorously [7, 16]. I this cotext, the Decisio Support Systems (DSS), seem to be useful for the evaluatio of the qualificatios i the uiversities rakig. I this paper, we shall focus o the area of idustrial egieerig withi the Spaish uiversity system ad o the structure of the Database as a fudametal elemet of the DSS. The aim of the paper is the use of the fuzzy AHP process to obtai the weight of the criteria withi a DSS i order to 2 The ANECA Process I this sese ad withi the framework of the Istitutioal Assessmet Programme (Programa de Evaluació Istitucioal, PEI ), the Spaish Natioal Agecy for Quality Assessmet ad Accreditatio (Agecia Nacioal de Evaluació de la Calidad y Acreditació ANECA ) presets this Guide [2] with the purpose of showig the steps i the Process of Istitutioal Assessmet The primary objective of the Istitutioal Assessmet Programme is to facilitate a assessmet process to officially improve the quality of educatio leadig to obtai uiversity degrees throughout the atioal territory, through self-diagosis ad the exteral view brought by experts. The developmet of this programme is iteded to promote assessmet processes favourig the establishmet or cotiuity of processes guarateeig quality i teachig, as well as providig iformatio to the studets ad their families, to society, to the goverig bodies of the uiversities ad to public admiistratios, regardig the quality of uiversity teachig ad their actio plas. This process is orgaised i three phases, see figure 1: Self-assessmet: the Self-assessmet Report describes ad evaluates the situatio of the assessed degree with respect to the criteria established, idetifyig stregths ad weakesses ad ehacemet proposals formig the basis of executio of the actio plas that must be iitiated o coclusio of the etire process. The report is writte by the Self-assessmet Committee. 675
2. T X is the term set of X, that is, the collectio of its liguistic values 3. U is a uiverse of discourse, 4. G is a sytactic rule for geeratig the elemets of TX ad 5. M is a sematic rule for associatig meaig with the liguistic values of X. Figure 1: Phases of the Istitutioal Assessmet Programme Exteral Assessmet: a group of exteral assessors to the teachig istitutio assessed, appoited by ANECA, ad uder its guidelies ad supervisio, aalyzes the Self-Assessmet Report, both through the study of documets ad by meas of a visit to the uit assessed, issues its recommedatios, ad proposes improvemets. The result of this phase is the Exteral Assessmet Report. Pla for improvemet: it collects the mai results of the Assessmet process. This phase cocludes with the pla for improvemets of the degree, describig the proposed improvemet actios i the Self-assessmet ad Exteral Assessmet phases, oce their viability is aalyzed. The tasks to be performed are determied accordig to their accomplishmet, the persos resposible, resources ivolved, deadlies for their implemetatio, the idicators moitorig the actios proposed, ad the beefits expected from them. 3 Liguistic variable ad fuzzy sets 3.1 Liguistic variable Most of the times, the decisio-maker is ot able to defie the importace of the criteria or the goodess of the alteratives with respect to each criterio i a strict way. I may situatios, we use measures or quatities which are ot exact but approximate. Sice Zadeh [19] itroduced the cocept of fuzzy set ad subsequetly wet o to exted the otio via the cocept of liguistic variables, the popularity ad use of fuzzy sets has bee extraordiary. We are particularly iterested i the role of liguistic variables as a ordial scale ad their associated terms, i this case triagular fuzzy umber, as used i the multi-criteria decisio makig. By a liguistic variable, [20,21,22], we mea a variable whose values are words or seteces i a atural or artificial laguage. For example Age is a liguistic variable if its values are liguistic rather tha umerical, i.e., youg, ot youg, very youg, quite youg, old, ot very old ad ot very youg, etc., rather tha umbers as 20, 21,22, 23,.... Defiitio.1- A liguistic variable is characterized by a quituple X ; T X ; U; G; M i which 1. X is the ame of the variable, I geeral for the decisio-maker it is easier whe he/she evaluates their judgmets by meas of liguistic terms. I those cases, the cocept of fuzzy umber is more adequate tha that of real umber. 3.2 Fuzzy sets The we have idetified the liguistic variable with a fuzzy set [3,10,11]. The fuzzy set theory, itroduced by Zadeh [19] to deal with vague, imprecise ad ucertai problems has bee used as a modellig tool for complex systems that ca be cotrolled by humas but are hard to defie precisely. A collectio of objects (uiverse of discourse) X has a fuzzy set A described by a membership fuctio 0,1. f A with values i the iterval f A : X 0,1 f x Thus A ca be represeted as A A x X. The degree that u belogs to A is the membership fuctio f A x. The basic theory of the triagular fuzzy umber is described i Klir [12]. With regard to the fuzzy umbers, we will show oly the mathematical operatios that will be used throughout the developmet of the paper. Defiitio 2. If T 1 ad T 2 are two triagular fuzzy umbers defied by the triplets a 1, b1, c1 ad a 2, b2, c2, respectively. The, for this case, the ecessary arithmetic operatios with positive fuzzy umbers are: a) Additio T T a a, b b, c c (1) 1 2 1 2 1 2 1 2 b) Subtractio T1 T2 T1 T2 whe the opposite T c, b, a 2 2 2 2 the T1 T2 a1 c2, b1 b2, c1 a2 (2) c) Multiplicatio T T [ a a, b b, c c ] (3) d) Divisio 1 2 1 2 1 2 1 2 a b c T1T2 a1, b1, c1 1/ c2,1/ b2,1/ a2, 0,, 2 2 2 e) Scalar Multiplicatio 1 1 1 1 (4) kt ka, kb, k c (5) 676
4 The Aalytic Hierarchy Process Method (AHP) The Aalytic Hierarchy Process (AHP methodology [14,15] has bee accepted by the iteratioal scietific commuity as a robust ad flexible multi-criteria decisio makig tool for dealig with complex decisio problems. AHP has bee applied to umerous decisio problems such as eergy policy [1,7], project selectio [6], measurig busiess performace [1], ad evaluatio of advaced maufacturig techology [4,5]. Basically, AHP has three uderlyig cocepts: Structurig the complex decisio problem as a hierarchy of goal, criteria ad alteratives, Pair-wise compariso of elemets at each level of the hierarchy with respect to each criterio o the precedig level, ad fially vertically sythesizig the judgemets over the differet levels of the hierarchy. AHP attempts to estimate the impact of each oe of the alteratives o the overall objective of the hierarchy. I this case, we oly apply the method i order to obtai the criteria s weights. We assume that the quatified judgemets provided by the decisio-maker o pairs of criteria (C i, C j ) are represeted i a x matrix as i the followig: C 1 C 2 C C1 c11 c12 c1 C 2 c21 c22 c2 C........ C c 1 c2 c (6) The c 12 value is supposed to be a approximatio of the relative importace of C 1 to C 2, i.e., c 12 (w 1 /w 2 ). This ca be geeralized ad the statemets below ca be cocluded: 1. c ij (w i /w j ) i,j=1, 2,, 2. c ii =1, i=1,2,, 3. If c ij =, 0, the c ji =1/, i=1,2,, 4. If C i is more importat tha C j the c ij (w i /w j )>1 This implies that matrix A should be a positive ad reciprocal matrix with 1 s i the mai diagoal ad hece the decisio-maker eeds oly to provide value judgemets i the upper triagle of the matrix. The values assiged to c ij accordig to Saaty scale are usually i the iterval of 1-9 or their reciprocals. It ca be show that the umber of judgemets (L) eeded i the upper triagle of the matrix are: L 1/2 (7) where is the size of the matrix C. The matrixes associated to the AHP approach are reciprocal, thus: The maximum eigevalue ( max ) is a positive real umber ad such that max. Associated with this eigevalue is a vector whose compoets are also positive. If this vector is ormalized the vector of weights associated with the matrix is obtaied. Where the values are fuzzy, ot crisp; istead of usig lambda as a estimator to the weight, we will use the geometric ormalized average, expressed by the followig expressio: w i j1 m i1 j1 a, b, c ij ij ij a, b, c ij ij ij where, aij, bij, cij is a fuzzy umber. 5 The database The Database has two parts. The first part cosists of the summary of the differet Reports of Exteral Evaluatio, published by ANECA. These reports are give by the members of the Exteral Assessmet Committee who are persos qualified by ANECA; the valuatio give by the exteral assessors is impartial. The secod part of the database, which is the focus of this paper, correspods with obtaiig the weights of the criteria, as well as the umerical represetatio of the labels. See Figure 2. Figure 2: The database correspodig to this part of the work. The study carried out has bee based o a questioaire desiged to such effect. ANECA itself set the questioaire to the experts that it determied as proper assessors. The questioaire was set / aswered by e-mail. This questioaire cosists of two clearly differetiated parts; The first oe has bee produced o the basis of the hierarchic structure of the criteria ad subcriteria (Table 1), usig the AHP methodology to do so. The secod part refers to the liguistic labels used i the ANECA survey for the exteral evaluatio of uiversity qualificatios. (8) 677
Table 1. Weightig criteria/subcriteria CRITERIA Geeral Weightig 1st LEVEL 2d LEVEL 3rd LEVEL 3th. LEVEL 2d LEVEL 1st LEVEL 1.1. Aims of the programme Aims of the E.P. (3.39,5.26,8.01) Admissio profile (3.39,5.26,8.01) (6.77,10.52,16.02) 1. Educatioal Curricular cotet (1.50,2.32,3.54) programme Curricular coherece (1.50,2.32,3.54) (E.P) 1.2. Studies pla ad its structure Curricular cosistecy (1.50,2.32,3.54) Curricular updatig (1.50,2.32,3.54) Aims of E.P. (1.50,2.32,3.54) (7.49,11.61,17.72) (16.61,22.13,29.14) 2.1. Maagemet ad plaig Plaig (5.27,7.94,11.81) (5.27,7.94,11.81) 2. Orgaizatio of teachig 3. Huma 4. Material 5. Traiig Process Commuicatio (1.59,2.39,3.57) 2.2. Maagemet ad Orgaisatio of teachig (1.59,2.39,3.57) orgaizatio Improvemet processes (1.59,2.39,3.57) (4.78,7.16,10.70) 3.1. Academic staff (AS) Appropriateess AS (5.11,7.95,12.15) Implicatio AS (5.11,7.95,12.15) (10.21,15.90,24.29) 3.2. Admiistratio ad service staff (ASS) Adaptatio ASS (2.50,3.79,5.94) (2.50,3.79,5.94) 4.1 Classrooms Appropriateess for umbers of studets (1.58,2.50,3.94) (1.58,2.50,3.94) Appropriateess for umbers of 4.2 Work Spaces studets (0.51,0.81,1.31) Appropriateess (AS ad ASS) (0.51,0.81,1.31) Ifrastructures: practical (0.51,0.81,1.31) (1.53,2.44,3.93) 4.3. Laboratories, workshops ad Appropriateess for umber of experimetal spaces studets (1.82,2.86,4.53) (1.82,2.86,4.53) 4.4. Library ad documet baks Correctly furished (0.69,1.09,1.78) Quality, quatity, (0.69,1.09,1.78) (1.38,2.19,3.56) Capture (0.54,0.84,1.33) 5.1. Studet assistace ad itegral traiig 5.2. Teachig-learig process Studet welcome actios (0.54,0.84,1.33) Support programmes (0.54,0.84,1.33) Professioal orietatio programmes (0.54,0.84,1.33) Tutorial actio programmes (0.54,0.84,1.33) Itegral traiig (0.54,0.84,1.33) Methodology (1.55,2.46,3.84) Evaluatio (1.55,2.46,3.84) Exteral practical (1.55,2.46,3.84) Mobility (1.55,2.46,3.84) (3.24,5.06,8.01) (6.21,9.84,15.34) (11.33,15.10,20.04) (14.44,19.70,26.70) (7.55,9.99,13.46) (11.04,14.90,20.10) 6. Results 6.1. Results of educatioal programme 6.2. Graduate results Effectiveess of E.P. (1.56,2.73,4.67) Studet satisfactio (1.56,2.73,4.67) (3.13,5.47,9.35) Compliace with the graduate profiles (3.39,5.99,10.34) (3.39,5.99,10.34) 6.3. Academic staff results Academic staff satisfactio (1.82,3.21,5.70) (1.82,3.21,5.70) 6.3. Results i society Employers ad other groups (0.10,1.76,3.08) Social lik (0.10,1.76,3.08) (0.20,3.52,6.17) (8.54,18.18,24.75) Source: ow productio 100.00 100.00 100.00 Likewise, we will take ito accout the possible valuatios of the experts, expressed i liguistic terms (A, B, C, D) These labels are arraged from largest to smallest as follows: A > B > C > D ad with the sematics that we will see later. It should be remembered that the results obtaied correspod to a problem of group decisio-makig, formed by experts, from whom we will obtai: 1.-The criteria weight as a result of the cosesus 2.- The membership fuctios of the fuzzy umbers that represet the liguistic labels A,B,C ad D. 5.1 Obtaiig the weightigs of the criteria/subcriteria Accordig to Table 1, the hierarchy structure has three levels. Cosiderig expressio (7), the large umber of questios might lead to the survey ot beig aswered, sice L = 74, as we have 6 i 1 st level criteria, subcriteria 16 i 2 d level ad 37 i 3 rd level idicators. This results i a questioaire which is ot feasible for experts to aswer. For this, ad sice the ifluece of the third level o the weight of the previous oes is practically ull, a uiform distributio o them was supposed. 678
For the first ad secod level, the procedure is as follows: firstly, we ask if all the criteria/subcriteria have the same weight, if so, we pass to the followig level. O the cotrary, if they do ot have the same weight, it will be cotiued by the followig questio of the questioaire. I this part, there is a questio that askig about the order of importace of the criteria/subcritera; ad fially, usig the liguistic labels defied by Saaty [14], the criteria/subcriteria are compared at the same level, takig ito accout the order established before. We chose to ask oly for oe row of the pairwise matrix ad from here to geerate the rest of the iformatio matrix, which was carried out completely cosistetly. Thus, this part of the questioaire had oly 21 questios. 5.2 Obtaiig the liguistic evaluatios The secod part of the questioaire is based o the semiquatitative survey of the Evaluatio of the Educatio iside the protocol to elaborate the Report of Exteral Evaluatio developed by ANECA, i which the followig labels are used: A: Excellet, B: Good, C: Average, D: Deficiet. For the valuatio of these labels we use the iterval [0, 10]. These liguistic variables, by the ucertaity of their ature, justified the use of fuzzy umbers associated with each liguistic term. whe: I A S A 1 S A 1 1 S A, i M i R i L i I this way, we have defied a fuzzy umber as a S A, S, fuctio of the three itegrals, L i where RA i S A ad M i R A i S represets the upper mea value associated R with the iverse fuctio of f ( x ), S A L A i is the lower mea value of the g (x) fuctio ad S A is the area of L A the core of the fuzzy umber, [0,1] is the idex of modality that represets the importace of the cetral value agaist the extreme values, ad [0,1] is the degree of optimism of the decisio maker. We have cosidered the case i which the three areas have the same weight ad it would correspod to the eutral decisio maker, whe 12 y 13 Takig as a example (Table 3), the outcome results of the DSS, for the real case of the evaluatio of five uiversities i the qualificatios of idustrial egieerig, where we have obtaied results both for the pricipal criteria ad for the global evaluatio. Table 3: Rakig result for five Uiversities. C1 C2 C3 C4 C5 M i C6 IAP 6 The results of the aggregatio for the group decisio experts By meas of a primary group decisio-makig process, it is possible to see the results of the weightig of the criteria ad sub-criteria for all the experts (Table 1). Table 2. Fuzzy umbers associated with the labels A,B,C ad D. Semi quatitative Geeral labels A: Excellet (8.1354, 9.4054, 10.0000) B: Good (5.8108, 7.1081, 8.4054) C: Average (3.5090, 4.8108, 6.1126) D: Deficiet (0.7355, 2.5135, 4.2916) Now, takig these weightigs as the base; ad the defiitio of semiquatitative labels (Table 2), obtaied by ANECA s ow experts, it is possible for us, attedig to these six criteria, to order the differet Uiversities. We make referece, i this case, oly to the area of idustrial egieerig. For the evaluatio of the alteratives the methodology used has bee the fuzzy weighted sum model as: FWSM w j a ij (9) j1 where, w j ad a ij are fuzzy umbers. The defuzzificatio method used is described i [8] Criteria Educatioal Programme Teachig Orgaizatio Huma Material Educatioal Process Results Gobal U1 0.062 0.033 0.071 0.040 0.042 0.056 0.306 U2 0.100 0.067 0.106 0.049 0.055 0.089 0.466 U3 0.129 0.086 0.105 0.067 0.098 0.119 0.604 U4 0.118 0.080 0.088 0.038 0.075 0.081 0.482 U5 0.125 0.089 0.105 0.047 0.082 0.100 0.548 7 Coclusios For the assessors/experts it is simpler to express their kowledge by meas of liguistic labels, istead of havig to do so by meas of umerical values. For that reaso, it is preferable to prepare a questioaire to obtai the experts kowledge, i which the aswers are i the form of liguistic variables. These liguistic variables have bee modelized by meas of fuzzy triagular umbers ad from a methodology widely accepted by the scietific commuity, sice it is the Aalytical Hierarchic Process; developed by Saaty i 1980. Takig Table 1ito accout, we coclude that i geeral the most importat criterio is the Educatioal Programme ad that the lowest weightig is obtaied for the Material. For future work, it would be iterestig to carry out a study of the aggregatio of the iformatio as a secodary process. I this process the weight of the criteria ad subcriteria are obtaied for every expert. Later all this iformatio would be aggregated. Moreover, a comparative primary ad secodary study of both types of processes would be desirable. 679
Similarly, as future work, it would be iterestig to make a comparative study of the evaluatio criteria accordig to the area of the differet experts. It is possible that the valuatio of a expert from the area of egieerig does ot value the criteria i the same way as aother from the humaities. Ackowledgmet This work is partially supported by the DGICYT uder project TIN2008-06872-C04-04 ad Juta de Adalucía (P07-TIC02970), respectively. Refereces [1] K. M. A. Al Harbi. Applicatio of the AHP i project maagemet. Iteratioal Joural of Project Maagemet;,19(1), 19-27, 2001. [2] Agecia Nacioal de Evaluació de la Calidad y Acreditació (ANECA) (2007). Programa de Evaluació Istitucioal. Aualidad 2006/2007. Guía de evaluació extera 2007. http://www.aeca.es/active/active_ese_pei0607.asp [3] R.E. Bellma, L.A. Zadeh. Decisio-makig i a fuzzy eviromet. Maagemet Sci., 17, 141-164.,1970. [4] Cha F.T.S, Cha M.H, Tag N.K.H. Evaluatio methodologies for techology selectio. Joural of Materials Processig Techology 2000a ; 107: 330 337. [5] F.T.S Cha, B.Jiag, N.K.H. Tag. The developmet of itelliget decisio support tools to aid the desig flexible maufacturig systems. Iteratioal Joural of Productio Ecoomics, 65 (1): 73 84, 2000. [6] C.-H. Cheg, K.-L. Yag, C.-L. Hwag. Evaluatig attack helicopters by AHP based o liguistic variable weight. Europea Joural of Operatio al Research 116 (2),423 435, 1999. [7] G. Federkeil.. Rakigs ad Quality Assurace i Higher Educatio. Higher Educatio i Europe, 33, 2/3, 219-230, 2008. [8] M.S, Garcia-Cascales,., M.T. Lamata,: A modificatio to the idex of Liou ad Wag for rakig fuzzy umber, Iteratioal Joural of Ucertaity, Fuzziess ad Kowledge-Based Systems, 411-424, 2007. [9] M. M. Kabla. Decisio support for eergy coservatio promotio: a aalytic hierarchy process approach. Eergy Policy, 32 (10): 1151-1158, 2004. [10] J. Kacprzyk, R.R. Yager. Liguistic summaries of data usig fuzzy logic. Iteratioal Joural of Geeral Systems. 30, 2, 133-154, 2001. [11] E.E. Kerre. The use of fuzzy set theory i electrocardiological diagostics, i: M.M. Gupta ad E. Sachez, Eds., Approximate Reasoig i Decisio Aalysis Amsterdam; North-Hollad 277-282, 1982. [12] G.J. Klir ad B. Yua, Fuzzy sets ad Fuzzy Logic, Pretice Hall PTR, New Jersey e d. 1995 [13] P. Motesios, J. M.Carot, J. M. Martiez, F. Mora. Third Missio Rakig for World Class Uiversities: Beyod Teachig ad Research. Higher Educatio i Europe, 33, 2 & 3, 259 27, 2008. [14] T.L. Saaty. The Aalytic Hierarchy Process. Macgraw Hill. 1980. [15] T.L. Saaty. Group Decisio Makig ad the AHP. Spriger Verlag. New York. 1989. [16] J. Sadlak J. Merisotis, N.C. Liu. Uiversity Rakigs: Seekig Prestige, Raisig Visibility ad Embeddig Quality. Higher Educatio i Europe, 33, 2 & 3, 195 199, 2008. [17] M.E. Smith. Chage ad iovatio i higher educatio: a role for corporate strategy?. Higher Educatio 16:37-52, 1987. [18] E. de Weert. A macro-aalysis of quality assessmet i higher educatio. Higher Educatio 19:57-72, 1990. [19] L. A. Zadeh. Fuzzy sets. Iformatio ad Cotrol, 8:338-353, 1965. [20] L. A. Zadeh. The cocept of a liguistic variable ad its applicatio to approximate reasoig: Part 1. Iformatio Scieces, 8, 199-249, 1975. [21] L. A. Zadeh, J. Kacprzyk (eds). Computig with Words i Iformatio / Itelliget Systems 1. Foudatios. Studies i Fuzziess ad Soft Computig, Physica-Verlag (Spriger-Verlag), Heidelberg ad New York. Vol. 34 1999. [22] L. A. Zadeh, J. Kacprzyk (eds). Computig with Words i Iformatio / Itelliget Systems 2. Applicatios. Studies i Fuzziess ad Soft Computig. Physica-Verlag (Spriger-Verlag), Heidelberg ad New York. Vol. 33 1999. 680