MODELING ITEM RESPONSE DATA FOR COGNITIVE DIAGNOSIS

Transcription

1 184 1st International Malaysian Educational Technology Convention MODELING ITEM RESPONSE DATA FOR COGNITIVE DIAGNOSIS Suhaimi Abdul Majid, Norazah Mohd. Nordin, Mohd Arif Hj. Ismail, 1 Abdul Razak Hamdan Faculty of Education Universiti Kebangsaan Malaysia Bangi, Selangor, Malaysia kam1676@hotmail.com, drnmn@ukm.my, mdarif@ukm.my 1 Faculty of Information Science & Technology Universiti Kebangsaan Malaysia Bangi, Selangor, Malaysia arh@ftsm.ukm.my ABSTRACT Standardized testing and classroom testing are common ways used in educational assessment. A single test score or grade is usually used to report student s mastery, which is of a normreferenced type. However, students scores or grades in a test lack cognitive information such as the details of their mastery. Researchers today are constantly looking for ways to develop assessments beyond the norm-referenced rank-ordering functionality. Currently, there are distinguished research findings in the field of cognitive psychology and psychometrics. Many tremendous developments in information and communication technology (ICT) have also emerged. Cognitive diagnosis is one of the recognized ways in cognitive psychology that has relation with psychometrics. With the integration of these two fields and powered with highly computational capability, a profile of knowledge, skills and proficiencies acquired by each learner can be detected from the item response patterns. The purpose of this paper is to give an insight on some of the cognitive diagnosis models which derived from item response data. The paper also reveals the strengths and limitations of the models from the perspective of statistical method and data mining technique. The potential of cognitive diagnosis models to be integrated in the ICT-based learning concludes the discussion of the paper. INTRODUCTION Educational assessment is an instructional process that plays a critical role to provide teachers with information about students learning. Testing is a popular way to assess students learning and a single test score is typically used to report student s mastery. But, students scores in a particular test lack cognitive information such as details of their mastery in the tested domain. Detailed descriptions of cognitive processes underlying test scores are useful since this information provides feedback for students and teachers to improve students learning performance. Teachers can also use this information to remediate the needy students and monitor the progress of individual s performance (Kim, 2004 & Cui et al., 2006). Data in standardized testing and school testing are abundant now and are stored whether in databases or manual records. Since the era of Classical Test Theory, many studies have been conducted to elicit knowledge from examinees responses to the test items. Consequently, many statistical models have evolved. Data mining is another potential field in ICT whereby its techniques are applied in many areas to acquire hidden knowledge from enormous data. The purpose of this paper is to give an insight into some cognitive diagnosis models that have been explored to diagnose students knowledge states. The described models are derived from the dichotomous responses to the test items. The strengths and limitations of the models from the perspective of statistical method and data mining technique are also exposed. This paper concludes with the potential of the models to be integrated in the ICT-based learning.

2 185 1st International Malaysian Educational Technology Convention CURRENT PRACTICES IN EDUCATIONAL ASSESSMENT Formerly, the behaviorist view was the dominant learning theory. At present, it has changed to a more integrated approach of cognitive and constructivist learning theories. The connection of prior knowledge to shape new learning and the importance of deep understanding to support knowledge transfer are among the characteristics of that integrated theory. As the result is the generally referred to as modern educational assessment which no longer focus on assessment with objective tests of limited scope, but towards both improving students learning and evaluating teaching. The emphases of this assessment are on what students can do, what skills they possess, and what problems they can solve. An on-going classroom assessment that is integrated with instruction, comprehensively integrated and combined skills and knowledge, as well as having multidimensional scores of meaningfully diagnostic values are among the characteristics of this modern educational assessment (Payne, 2003). Malaysia too is in the process of reforming the assessment system towards improving the performance of both students as well as teachers. School-based assessment has been carried out in Malaysian schools since This new yardstick of assessment is aimed to balance between the summative assessment and formative assessment. There is also a need for a coherent assessment system that provide conceptual link between goals, instruction, measures and subsequent actions. To accomplish those needs, an ICT-based system that is able to track students progress and provide holistic report in terms of cognitive, affective and psychomotor abilities is highly in demand (Malaysia Examination Syndicate, 2006). In short, the current practice of educational assessment is in the transition period from using assessment solely for assigning grades to a broader view of assessment as a process of gathering valid and relevant evidence about what students know (Rupp, 2006). Researchers in current assessment practices emphasize the need to integrate assessment into the curriculum and instruction process, so that the assessment is more formative and more prescriptive or diagnostic. New kind of mechanism is needed to provide valuable feedback to teachers and help teachers to identify what a student has learned, as well as to decide how instruction needs to be adapted to the students (Ye, 2005). INTEGRATION OF PSYCHOMETRIC MODELS, COGNITIVE DIAGNOSES AND ICT Over the past century and more, the field of psychometrics has developed ways of measuring the latent quantities of knowledge and ability in the human mind or commonly known as latent traits in psychometrics. Classical Test Theory (CTT) and Item Response Theory (IRT) are among the earliest statistical approaches and universally used in determining the latent trait of an examinee possessed through the testing data (Stout & Hartz, 2004 & Baker, 2001). The idea behind CTT is that the observable test score is not the value representative of an examinee s performance on the test. The observed test scores in a test is the composition of true and error scores. The focus of the theory is on the properties of test scores relative to the populations of examinees and not to analyze individual test scores. CTT can be used to assess the quality of test scores, it does not provide efficient model-based tools for estimating latent variables (Sijtsma & Junker, 2006). Item Response Theory (IRT), also known as latent trait theory, has been applied more frequently as a foundation theory in educational and psychological measurement research. IRT model is a mathematical function that has relationship between observable performance and certain latent traits or abilities. The purpose of IRT model is to estimate examinees abilities measured by a test. The test item is the unit of measure used to obtain ability scores on the same scale. The IRT models the probability of a correct response to a test item as a function of one or more parameters of the item and the latent trait level of the examinee. As the trait level increases, the probability of responding to the item correctly will also increase.

3 186 1st International Malaysian Educational Technology Convention Three most commonly used dichotomous IRT models are the one-parameter logistic model (1PL), the two-parameter-logistic model (2PL) and the three-parameter logistic model (3PL). The 1PL model or also referred as the Rasch model considers the difficulty level of the items on the test. The higher the difficulty level, the harder the examinee will respond to, and hence need a higher level of the ability trait. The 2PL model deals with item difficulty and item discrimination. Item discrimination reflects an item s facility in discriminating between examinees of different ability levels. Finally, the 3PL model includes both the difficulty level and discrimination parameter, but also adds the pseudo-chance level or guessing parameter. It is assumed that the examinees have some non-zero probability of responding correctly to the item regardless of ability level that make the guessing parameter as part of the 3PL model (Kim, 2004). One important assumption of IRT models is that only one kind of dominant factor or latent trait affects the test performance. This means that test items measure only one kind of ability. Due to this limitation, researchers today are continuously looking for new assessments that include criterion-referenced diagnostic capacities so that stakeholders have detailed information about learner characteristics and assessment properties and that are beyond the norm-referenced rankordering functionality. Developments of such assessments have also been fostered by recent research findings in cognitive psychology, advances in psychometric research and rapidly increasing capability of computer systems in the 1990s (Rupp, 2007 and delafayette Winters, 2006 ). Cognitive diagnosis is one application of psychometric theory which can be applied in educational assessment. It is sometimes referred to as skills diagnosis, skills assessment, skills profiling or profile scoring. It statistically evaluates the level of competence on an array of skills for each examinee. It also evaluates the skills estimation effectiveness of a test by assessing the strength of the relationship between the individual skills profiles and the observed performance on individual test items (Roussos et al., 2007). Ohlsson (1986) defines cognitive diagnosis as the process of inferring person s cognitive state from his or her performance. It is a central activity in any system which aims to build a dynamic model of the user of that system. An intelligent computer system is one of the emerging new technologies that can make possible to gather, capture and analyze data to give the true depth of students understandings and multidimensional information. Embedding a cognitive diagnosis model in a computer-based diagnostic assessment system or intelligent tutoring system is one way of accomplishing those tasks (Kim, 2004). In the intelligent tutoring system, cognitive diagnosis is often used synonymously with student modeling to denote the process of building representation of the student s knowledge from the evidence provided by student inputs to solve problem (Surgeiva & Khan, 2004). Nowadays, new cognitive diagnosis model can even make explicit the test developer s actual assumptions regarding processes and knowledge structures a student in a test domain would use, how the processes and knowledge structures develop, and how more competent students differ from less competent students (Cui et al., 2006). STATISTICAL MODELS OF COGNITIVE DIAGNOSIS For this paper, statistical model is meant to stand for mathematical equations that formalize probabilistic processes and whose parameters can be estimated with appropriate routines for real and simulated data sets under certain required assumptions (Rupp, 2007). IRT-based statistical methods are currently heavily used statistical methods to assess examinee latent traits ability levels (Stout & Hartz, 2004). Statistical models for cognitive diagnosis are classified according to their purpose identified by Hartz, Roussos and Stout (2002). Two purposes of cognitive diagnosis modeling are determining the examinee s profile of mastery or non-mastery attributes in a test and evaluating the effectiveness of a test and its items in measuring the individual attribute. Thus, it is either an examinee-based model or item-based model or unified of both targets. One additional characteristic to complement the model as an ideal cognitive diagnosis model is the ability to statistically estimate the parameters involved in the model.

4 187 1st International Malaysian Educational Technology Convention Fischer s linear logistic test model (LLTM) is an important early model of the cognitive diagnosis that has connection with an item score (Fischer, 1973). The groundwork of LLTM has been the basis for an item-based cognitive diagnosis model. Tatsuoka s Rule Space Methodology (Tatsuoka, 1983) is the early foundation for the examinee-based cognitive diagnosis model. Based on these two models, a Unified Model (DeBello et al, 1995) was developed, and then a Fusion Model (Hartz et al., 2002) appeared that overcame the limitations of most cognitive diagnostic models. There are many other IRT-based cognitive diagnosis models, but the above models are mostly described and have been the basis for the derivation of other models. Fischer s Linear Logistic Test Model (LLTM) The LLTM is the expansion of the Rasch model. Rasch item difficulty, also called effect, is decomposed into discrete cognitive attribute-based difficulties. The item difficulty or effect is a linear function of the weighted sum of the attribute-based difficulties in LLTM. The examinee parameter in LLTM remains as a single unidimensional ability parameter. Hence, LLTM measures the probability for an examinee to response correctly on an item with a weighted sum of attributebased difficulties and the examinee s ability parameter on that item. The limitation of LLTM is that it lacks the measure for evaluation of individual examinees with respect to individual attributes since the examinee parameter remains a single unidimensional ability parameter. Second, the difficulty of an attribute indicates its difficulty across the entire test; not for each suitable item. The Rule Space Method (RSM) The RSM developed by Tatsuoka (1983) is a probabilistic approach whose purpose is to identify the examinee s knowledge state, based on an analysis of the task s cognitive requirements, also called task attributes. The knowledge state is represented in an attribute vector that combines attributes which are mastered and not mastered by an individual examinee. The first step in RSM begins with the identification of attributes for the test items of interest. Item attributes may include knowledge, strategies and processing skills that are required to answer the items correctly. The incidence matrix (or Q-matrix as it is well known) is then constructed. Q- matrix is binary and of order K rows x m columns. K is also the number of attributes to be measured and m is the number of items on the test. The values of 1s are assigned to attributes that have relationship with the items and 0s to those that are not related. The identification of attributes and the construction of Q-matrix are generated by domain experts. The second step is to determine the ideal item-response patterns. These ideal item-response patterns are established systematically by using rules and derived from Q-matrix. A rule is a set of procedures that one can use to solve a problem. Rules are determined through logical task analysis or deterministic methods used in artificial intelligence. A specific feature of Boolean algebra called Boolean description functions are used to determine the ideal item-response patterns. The assumption behind Boolean descriptive function (BDF) is that an item can be answered correctly if and only if the attributes involved in this item have been mastered. Based on the attributes involved in an item and applying BDF, each ideal item-response pattern can correspond to the observable item-response pattern. The observable item-response pattern is a vector of ones and zeros that represents how an examinee performed on a test. However, examinees observed item-response patterns are often different from the ideal itemresponse patterns due to using the rule inconsistently. Hence, the third step is to map these two patterns onto a two-dimensional classification space called rule space. A Cartesian Coordinate System with θ (IRT ability parameter) on the x-axis and ζ (the unusualness of item response pattern) on the y-axis is used for formulating the classification space. An unusual item response pattern is one which easy items are answered incorrectly while difficult ones are answered correctly. Values of ζ are calculated as the standardized product of two residual matrices, where residuals are the difference between observed and expected value and standardization is achieved by dividing the product by the standard deviation of its distribution. The last step is to classify examinee s item-response pattern into one or more of predetermined latent knowledge states. Mahalanobis distances between the examinee s item-response pattern

5 188 1st International Malaysian Educational Technology Convention and the ideal-response pattern are calculated in order to determine the classification group the examinee belongs to. The model generated from RSM can be illustrated visually, such as influence diagrams and knowledge state trees. An influence diagram shows which item measures which attributes and which attributes influence other attribute. A knowledge state tree portrays the incremental relationships between the knowledge states. This feature can guide examinee of how to improve from one knowledge state to a more knowledge state. Although RSM is one of the fundamental Cognitive Diagnosis theories, it has not been widely accepted for some limitations. The main drawback is that RSM does not provide any evaluation of the relationship between the items and the attributes. Uncertainties may also be the result from RSM. The Q-matrix constructed by human, such as the content experts may produce insufficient representation of attributes required by each item (Hartz, 2002). Further more, examinees might use different strategies to solve the items. This occurrence may require a different Q-matrix from what is derived by experts and RSM also has no way to measure or validate this deviation (Barnes, 2003). In fact, in a study using two separate Q-matrices on the same items has resulted in different difficulty measures for the same item (Birenbaum et al, 1993). The Unified Model The Unified Model, developed by DiBello, Stout and Roussos (1995) is based on RSM and integrated with LLTM. It decomposes examinees abilities into cognitive components and the item difficulty parameter into discrete attribute-based difficulties. Thus, the Unified Model has both features of item-based attribute parameters and examinee-based attribute parameters. The Unified Model contains the sources of systematic deviations from the response behavior predicted by the Q-matrix. As compared to RSM, there is one type of random error which is considered all systematic errors as random slips. In the Unified Model, the source of random error is broken down into four types of systematic error. First, the source of response variation is derived from strategy selection, because an examinee may use a different strategy for an item than the one assigned in the Q-matrix. Second, error might occur because the attributes specified for certain items in the Q-matrix are incomplete. This source of error is classified as completeness of the Q-matrix. Third, positivity as the source of random error. Positivity is defined as the inconsistency of examinees responses. An examinee who does not master a certain attribute might respond correctly to an item that measures the attribute; or an examinee who possesses an attribute but does not get correctly on the item that applies the attribute. Lastly, slips still remain as random errors. This category is for errors that cannot be explained by the above three categories, such as mistakes because of carelessness. An important advantage in using the Unified Model is that it introduces a parameter called the latent residual ability. The latent residual ability is the remaining latent ability which involved constructs that are not measured by Q-matrix. It also deals with the issue of different strategy used to solve an item. The latent residual ability is represented by another parameter in the Unified Model; not as additional attributes in Q-matrix. The Unified Model includes a large number of parameters for the model to deal with. The issue aroused is that the model must balance between improving the accuracy of measurement and statistical identifiability of the many parameters in the model. However, there are some parameters in the Unified Model that are not identifiable, such as the item parameters that need to be estimated for the model to be calibrated. A reduction in the parameter space is also required to make the model estimable (Roussos et al., 2007). The Fusion Model The Fusion Model developed by Hartz, Roussos and Stout (2002) retains the advantages of the Unified Model while reducing the number of parameters involved so that they are statistically

6 189 1st International Malaysian Educational Technology Convention identifiable. This reduced model, also referred to as the Reparameterized Unified Model (RUM) has k i + 2 parameters per item compared to the 2k i + 3 parameters per item in the original Unified Model (k i is the number of attributes required by item i). RUM consists of the Q-based item i difficulty and the discrimination parameter of the item i for the attribute k. The discrimination parameter is the proportional parameter representing the ratio of the likelihood of a correct answer given mastery versus non-mastery. If this parameter is getting closer to zero, the more discriminating item i is for attribute k. When most of this parameter for an attribute are closer to zero, the test is said to display high cognitive structure for that attribute, which is the indicative of a test that is well designed for diagnosing mastery on the attribute. The search for the dependency indicator of an item on the non-q attributes is still retained in the Fusion Model. This unique component is introduced in the Unified Model and does not present in other cognitive diagnosis model. The Rasch Model with the negative item difficulty is applied to determine that probability. The higher the value of item difficulty, the less the item depends on the non-q attributes. The range of 0 and 3 for item difficulty is chosen as the diagnostic information for the dependency of the item on the non-q attributes. The specified Q-matrix is highly lacking some other attributes when the item difficulty is equaled to 0. A hierarchical Bayesian modeling approach with a Markov Chain Monte Carlo (MCMC) algorithm is adopted to increase the capacity of RUM to fit the data and to simplify and improve the estimation procedures. The program Arpeggio has been developed for this purpose. Flexibility in the estimation of item parameters is required for the use of the Fusion Model due to the positive correlations that exist among the different sub-skills included on the test. Arpeggio also provides estimates the posterior probability of mastery for each of the examinees. This estimate is used to classify examinees into one of three groups for each skill, which is either masters, non-masters or indeterminate cases. DATA MINING TECHNIQUE FOR COGNITIVE DIAGNOSIS Data Mining and Educational Data Mining Han and Kamber (2006) define data mining as the task of discovering interesting patterns from large amount of data that is stored in all kinds of information repositories such as databases, World Wide Web and flat files. It is the process of finding trends and patterns in data. The objective of this process is to sort through large quantities of data and discover the hidden knowledge. That make data mining and knowledge discovery as terms that are used interchangeably. Data mining is the confluence of many disciplines, such as database system, statistics, artificial intelligence, machine learning, visualization and information science. Many techniques from other disciplines or the integration of techniques can be applied, depending on the data mining tasks, the kind of data to be mined and its application. Commonly techniques such as Neural Networks, Fuzzy/Rough Set Theory, Genetic Algorithms and Bayesian Networks are widely used in many data mining application, specifically information retrieval, pattern recognition, Web technology, business, bioinformatics and psychology. The rapid development in ICT-based learning technologies have made possible for collecting vast amounts of student profile data. These data are quite heterogeneous, coming from multiple sources and being expressed in different scales. Data such as web log files, interaction logs, time series data, text, dialogue and human judgment data are available through Intelligent Tutoring System, Learning Management System and other education-related data sources. Mining those data can extract new knowledge about how people learn and the knowledge discovered can be applied to develop new ways to support and advance human learning (Barnes, 2003). Due to the needs to mine complex data about educational situation, a new academic area called Educational Data Mining (EDM) exists whose researchers are mostly from the Intelligent Tutoring System (ITS) community. EDM research too has cohesion with the research from the Artificial Intelligence in Education (AIED) community. EDM focuses on the collection, archiving and analysis of data related to student learning and assessment and its research is often related to

7 190 1st International Malaysian Educational Technology Convention techniques in psychometrics and educational statistics. The analysis part needs tools from data mining techniques to gain deeper understanding of student learning, discover relationships among questions, and possibly develop deeper quantitative understanding of cognitive processes (Minaei-Bidgoli, 2004). EDM techniques mostly applied in the development of ITS. ITS is a computer based instructional system that attempt to determine information about student s learning status, and use that information to dynamically adapt the instruction to fit student s needs (Minaei-Bidgoli, 2004). One of the component in ITS architecture is the student model which is a cognitive model of each individual learner. It provides input to the pedagogical model in ITS, so that the right pedagogical decisions are made to the differing needs of each student. Therefore, sound cognitive diagnosis that integrates cognitive theory and instructional science are required for the development of student model component of ITS. This will form the data structures component in student model which is the information structures representing the knowledge, skills and abilities of the student as learner. Apart from that, the student model also comprises the inference engine that permits a diagnosis or suggests a prescription (Everson, 2004). Cognitive Diagnosis Based on Overlay Model Overlay model is the earlier basis for the development of student model and was introduced by Carr and Goldstein in An overlay model assumes that student knowledge is a subset of expert knowledge. This expert knowledge is usually constructed either as network representations or rule-based representations. Student is considered not mastering certain knowledge when their knowledge are mismatched with expert knowledge. However, researchers discovered that student beliefs do not always coincide with expert knowledge (Barnes, 2003). Due to that reason, Brown and Burton introduced a rule-based modeling system called Buggy in 1978 as an extension to the overlay model. This is a diagnostic approach which identifies student s misconceptions by means of bugs library. A bug is a variant of procedure which generates an error. An error occurs when there is a difference between expected behavior and observed behavior in the response to a posed problem. The bugs library is an extensive catalogue which contains precisely defined systematic errors or misconceptions made by students. This fixed library is based on a large empirical study and requires considerable experts effort to construct it. It is difficult to get a complete bugs and there are cases where some bugs have complex structures. The consequence is some errors made by students may not correspond to the fixed bugs from the library. For these reasons, Brown and VanLehn (1980) developed Repair Theory which is a theory of how knowledge errors are formed. It is postulated that students with incomplete knowledge will face to a situation where they do not know what to do. This situation is called the impasse. Impasse may occur at a number of places while they are solving a problem and the student tries to apply various strategies to overcome the impasse. These strategies are called repairs. Some repairs result in correct outcomes or still generate incorrect result and hence bugs still exist. Hence, VanLehn introduced a model in a system called Sierra (1982). Sierra used machine learning techniques to produce rule sets, which are predictions of procedures that student could learn during a lesson. These rule sets are passed to a solver, based on Repair Theory, which implements the rules to solve problems. The generated answers are diagnosed as bugs or not by an automated diagnosis tool called Debuggy. An advantage of model that is based on Repair Theory is that it is remarkably effective at predicting bugs in human learners. However, it is argued that the model based on the Repair Theory is a local problem-solving mechanism. The rule sets that usually based on heuristics may well-described one group of students, but may have semantic constraints in another group who are at the same level of understanding. Furthermore, this approach is assuming that students will follow the rule sets constructed by experts and often a heuristic-based model is not good enough to guide instruction.

8 191 1st International Malaysian Educational Technology Convention Neural Networks At present, theories of learning which are based on work in cognitive science, psychology and educational measurement, view students as active learners in constructing knowledge. This theoretical shift concerned with the mechanisms that distinguish learners in terms of their knowledge states or in terms of their knowledge representations, cognitive process, and strategies for solving problems. Apart from the IRT-based models, there are also promising techniques such as Neural Networks and Bayesian Inference Networks which can be applied in the next generation ITS student model and can achieve that new demands (Everson, 2004). Neural Networks method imitates the neuron functions in human brain and it is a pattern recognition method which learns from training networks on a set of exemplary data. The work by Harp (1995) and his colleagues represents one of the earliest applications of neural network technology to the student modeling problem, specifically with the Kohonen Feature Map technique. The Kohonen Feature Map is a self-organizing clustering task technique in an unsupervised neural network learning environment. The input vector to this network is the students responses to test items which can be both multiple-choice and fill-in-the blank questions. The repeated iterations of neural network feature will train the system to generate student models referred to as Models of the Universe of Student Knowledge (MUSK). MUSK is a general knowledge structure or a set of all possible knowledge states of students in the tested domain. It captures the capabilities of students of different mastery levels and also indicates learning paths from lower to higher levels. MUSK is used as student model in ITS and predictions about how students will perform on sets of problems stored in ITS can be generated through the inferring process using MUSK. The Kohonen Feature Map technique can also handle the incomplete and noisy data sets which then can still produce a reliable MUSK. The self-organizing clustering task that does not acquire the experts involvement to develop the student model is another advantage of MUSK. Further more, MUSK can be updated periodically during the tutoring cycle. Thus, MUSK approach that applied the neural network method is capable of supporting adaptive assessment, monitoring student progress, selecting and providing learning sequences, and providing student feedback (Everson, 2004). Bayesian Networks Developing student model with numerical approach which can handle uncertainty reasoning in learning environments has gained interest in many ITS researches for the last few years. Bayesian inference networks or Bayesian networks (BN) is such an example and the most broadly used approach (Gonzalez et al., 2006). The idea of a BN comes primarily from the theory of graphical models. BN is defined as probabilistic or causal relationships among variables in a directed acyclic graph. The graph consists of nodes and arcs. The nodes represent variables, which can be discrete or continuous. The directed arcs between pairs of nodes represent causal relationships between them. In a BN, nodes without incoming arcs, i.e. without parents, are called root nodes and nodes without outgoing arcs, i.e. without children, are called leaf nodes. Root nodes have marginal prior probability distribution and all the other nodes have conditional probability distributions. Based on the assumption of conditional independence, the conditional probability distribution for a random variable associated to a node, is specified by considering the probability of each of its state conditioned on the combination of the states of its parent nodes (Zhou et al., 2006). As in educational assessments, BN is frequently used to reason about students knowledge and skills given information about their performance in assessment. A BN is appropriate tool when an examination of the items in an assessment suggests that solving item requires at least a certain level of one or more a number of prespecified skills (Sinharay, 2006). In the graphical model structure, each node can represent a specific concept, skill or misconception, thereby allowing detailed knowledge assessment. Each node can gather evidence from complex relationships between other nodes to yield a probabilistic assessment of mastery. They can be learned from

9 192 1st International Malaysian Educational Technology Convention data or engineered by domain experts when data is not available. A combination of data based approach and experts based approach can also be used to develop a BN model (Vomlel, 2003). Some examples of BN based student modeling are OLAE, POLA and ANDES. OLAE (Online Assessment of Expertise)( Martin & VanLehn 1995) uses BN to observe student behavior in solving introductory physics problems and compute the probabilities that the student knows and uses each of the rules in a given knowledge domain. It generates a student model that consists of a rule-based program that reflects the way student computes answers to actual problems, both correctly and incorrectly. POLA (Probabilistic Online Assessment) extended the OLAE system. It turns the knowledge tracing of OLAE into a system of probabilistic reasoning to predict the rules known and path used by student in solving a problem. Finally ANDES determine the student prior probabilities of knowing set of knowledge elemental items (Gonzlez et al., 2006). Andes student model also supports prediction of students actions during problem solving. This is especially relevant in a domain where there can be multiple solutions to a problem, in which each solution can be implemented through different action orderings. Being able to predict at inferences and actions a student can perform in the context of a particular solution greatly improves the help that the system can provide when the student does not know how to proceed (Conati et al., 2002). CONCLUSION There are other statistical models and data mining based models have been developed for the purpose of cognitive diagnosis. This paper exposed some widely discussed models especially the models that were derived from the students responses on test items. It is hoped that these discussed models will give alternative to educationists and assessment practitioners a broader view of assessments as a process of gathering valid and relevant evidence about what students know, apart from the globally used IRT and CTT models. Computer Based Assessment System, Integrated Learning System, Learning Management System and Intelligent Tutoring System are some ICT-based learning environment that will gain benefits from these models which is the integration of recent research findings in cognitive psychology, psychometrics and ICT. These models and tools can be embedded in those instructional systems which will then provide users with useful, detailed and instantaneous feedback. The systems can also prescribe students with individualized and adaptive learning, thereby ensuring students to learn within their needs. This will allow students to work independently while the teachers can monitor the individual student s progress. With the aid of cognitive diagnosis models that can be implemented in instructional system or other kinds of information system, individual student s progress can be measured over time in a consistent and sustained manner. If this kind of system is implemented together with a certain policy such as the Malaysia school-based assessment, the individual student s growth in academic progress too can be systematically monitored over his or her school years. In short, the implementation of cognitive diagnosis models can be a powerful diagnostic tool which may have other impacts on curriculum practices, professional development and pedagogy. Subsequently, this may enhance the student s academic achievement. REFERENCES Baker, F. B. (2001). The Basics of Item Response Theory. ERIC Clearing House on Assessment and Evaluation. Barnes, T.M. (2003). The Q-matrix method of fault-tolerant teaching in knowledge assessment and data mining. Doctoral Thesis. North Carolina State University. Birenbaum, M., Kelly, A., & Tatsuoka, K. (1993). Diagnosing knowledge state in algebra using the rule-space model. Journal of Research in Mathematics Education, 24(5), Brown, J.S., & VanLehn, K. (1980). Repair Theory: a generative theory of bugs in procedural skills. Cognitive Science, 4, Brown, J.S., & Burton, R.R. (1978). Diagnostic models for procedural bugs in mathematical skills. Cognitive Science, 2, Conati, C., Gertner, A., & VanLehn, K. (2002). Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction, 12(4),

10 193 1st International Malaysian Educational Technology Convention Cui, Y., Leighton, J.P., & Zheng, Y. (2006). Simulation Studies for Evaluating the Performance of the Two Classification Methods in the AHM. crame/research.htm delafayette Winters, T. (2006). Educational data mining: Collection and analysis of score matrices for outcomes-based assessment. Doctoral Thesis. University of California, Riverside. Everson, H.T. (2004). Intelligent tutors need intelligent measurement, or the other way round. In Rabinowitz, W., Blumbrg, F.C., Everson, H.T. (Eds.), The design of instruction and evaluation. Affordances of using media and technology. London: Lawrence Erlbaum Associates, pp Fischer, G.H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica 37, Gonzalez, C., Burguillo, J.C., & Llamas, M. (2006). A qualitative comparison of techniques for student modeling in Intelligent Tutoring Systems. Paper presented at the 36 th ASEE/IEEE Frontiers in EducationConference, San Diego, CA, October. Han, J. & Kamber, M. (2006). Data Mining: Concepts and Techniques. San Francisco: Morgan Kaufmann Publishers. Harp, S. A., Samad, T., & Villano, M. (1995). Modeling student knowledge with self-organizing feature maps. IEEE Transactions on Systems, Man & Cybernatics, 25(5), Hartz, S., Roussos, l., & Stout, W. (2002). Skills Diagnosis: Theory And Practice. User Manual for Arpeggio software. ETS. Kim, S. (2004). An automated test assembly for unidimensional irt tests containing cognitive diagnostic elements. Doctoral Thesis. University of Texas at Austin. Malaysia Examination Syndicate. (2006). Humanising assessment. Speech by Minister of Ministry of Education Malaysia at the Kuala Lumpur International Conference on Assessment (KLICA 2006), Kuala Lumpur, Malaysia, May. Minaei-Bidgoli, B. (2004). Data mining for a web-based educational system. Doctoral Thesis. Michigan State University. Ohlsson, S. (1986). Some principles of intelligent tutoring. Instructional Science, 14, Payne, D.A. (2003). Applied Educational Assessment. Belmont: Wadsworth / Thomson Learning Roussos, L.A., DiBello, L.V., Stout, W., Hartz, S., Henson, R.A., & Temptlin, J.L. (2007). The Fusion Model skills diagnosis system. In Leighton, J.P., & Gierl M. J. (Eds.), Cognitive Diagnostic Assessment for Education, Theory and Applications (pp ). New York: Cambridge University Press. Rupp, A.A. (2007). The answer is in the qquestion: A guide for describing and investigating the conceptual foundations and statistical properties of cognitive psychometric models. International Journal of Testing, 7( 2), Sijtsma, K., & Junker, B. W. (2006). Item Response Theory: Past performance, present developments, and future expectations. Behaviormetrika 33(1), Sinharay, S. (2006). Model diagnostics for Bayesian networks. Journal of Educational and Behavioral Statistics, 31(1), Stout, W.F., & Hartz, S.M. (2004). Latent property diagnosing procedure. Tatsuoka, K.K.(1983). Rule Space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, VanLehn, K. (1982). Bugs are not enough: Empirical studies of bugs, impasses, and repairs in procedural skills. The Journal of Mathematical Behavior, 3, Vomlel, J. (2003). Two applications of Bayesian networks. Proceedings of Conference Znalosti 2003, Ostrava, Czech Republic, Ye, F. (2005). Diagnostic assessment of Urban Middle School student learning of Pre-algebra patterns. Doctoral Thesis. Ohio State University. Zhou, Z., Jin, G., Dong, D., & Zhou, J. (2006). Reliability analysis of multi systems based on Bayesian networks. Proceedings of the 13 th Annual IEEE International Symposium and Workshop on Engineering of Computer Based Systems, Potsdam, Germany, March,