Searching as learning: A systematization based on literature

Size: px
Start display at page:

Download "Searching as learning: A systematization based on literature"

Transcription

1 Search as Learning Special Issue Searching as learning: A systematization based on literature Journal of Information Science 2016, Vol. 42(1) 7 18 Ó The Author(s) 2016 Reprints and permissions: sagepub.co.uk/journalspermissions.nav DOI: / jis.sagepub.com Pertti Vakkari School of Information Sciences, University of Tampere, Finland Abstract The paper surveys empirical studies on the relations between information searching and learning, and presents some reflections about learning in a search process based on the findings. First, the meaning of the concepts learning and searching is briefly defined. Learning is conceptualized as changes in one s knowledge structures. Then it is described more in detail how learning occurs in the search process. The point of departure is to focus on tasks that require the restructuring of knowledge structures and to analyse how gradual stabilization of those structures is related to accessing and interacting with information sources. After that, empirical studies on searching and learning are categorized by identifying independent and dependent variables in those studies. In conclusion, some general remarks on the topic are presented. Keywords Criteria of success; information searching; information retrieval; knowledge structure; learning; research type; search outcome 1. Introduction Some research in information science either conceptualizes searching as learning or otherwise explores links between searching and learning. Most of the studies belong to the latter category. These studies do not explicitly use the word learning, although implicitly they deal with this phenomenon. The major framework in information science conceptualizing searching as learning is Kuhlthau s information search process (ISP) model [1]. Although she speaks about the information search process, the model deals with learning tasks, and it can be generalized to a certain extent to all tasks requiring learning. According to Kuhlthau [1], the bibliographic paradigm describes the task of information seeking primarily as the gathering and collecting of information, rather than a series of tasks within a constructive (i.e. learning) process. Kuhlthau [1] differentiates between two major aspects of information seeking, (a) access to sources and (b) guidance in the construction process. Both aspects are interrelated and users need intervention support for both of them. Source-related interventions assist with access to information, whereas process-related interventions assist with learning through the use of information. In information science most of the studies relating to searching and learning belong to the bibliographic paradigm, because they focus on learning when accessing, but not when using information in the sources found. However, there is a growing interest in exploring also how users select, structure, manipulate and combine information in documents accessed for creating the outcome of a search [2, 3]. Designing and evaluating tools for interacting with information accessed is also gaining popularity [4]. Despite this interest, there is a lack of overviews surveying studies relating to searching and leaning. The aim of this essay is to characterize the concepts learning and searching, and in the light of these notions describe how learning occurs in the search processes. Based on this analysis the major types of studies about the relation between searching and learning are introduced. The article begins by defining the notions of learning and searching. Based on these definitions, it is characterized how learning occurs in the search process. It is suggested how various types of learning are associated with differing Corresponding author: Pertti Vakkari, School of Information Sciences, University of Tampere, Kalevantie 4, Tampere, FIN-33014, Finland. Pertti.Vakkari@uta.fi

2 Vakkari 8 stages in the search process. After that the major types of research on the relations between searching and learning are introduced by categorizing the independent and dependent variables in the studies. The conclusion includes implications for researchers wishing to analyse searching as learning. 2. Learning Learning is the act of gaining new or modifying and reinforcing existing knowledge (skill, behaviour, values), by studying, practising, being taught or experiencing something; or knowledge or skill gained from learning [5]. Learning theories can be categorized into three major groups, behavioural, cognitive and constructivist approaches. The respective conceptualizations are learning as response to strengthening, learning as information processing and learning as knowledge construction. More information about various notions of learning can be found in Ormrod [6]. The conception of learning selected for this study includes mainly gaining new or modifying and reinforcing existing knowledge. By knowledge is meant the totality what a person knows, that is, a personal knowledge or belief system. It may include both justified, true beliefs and less justified, not so true beliefs, which the person more or less thinks hold true [7]. Information is stored to some medium and communicated between two recipients. Information is transformed into knowledge in one s mind when it is processed and integrated into one s knowledge structures [7]. Studying information searching, treating learning as gaining knowledge, and perhaps as gaining skills, is more fertile than, for example, observing changes in values or behaviour in general. This idealization reduces the complexity of the phenomena of interest, and consequently, of analysis. As the definition above shows, learning occurs in many ways, not only by studying or by being taught, but also by experiencing something, like by doing. This definition includes both intentional and unintended learning. Learning occurs in all sectors of life, not only in education or schooling. All activities that are performed without sufficient knowledge (skills) require the acquisition of knowledge and learning to be performed. Naturally, the extent to which actors need new knowledge varies a lot depending on the task and their knowledge of the task. A novice securities analyst analysing how an industry is structured in a country needs to acquire much more information and process it to produce a report for a customer compared with an expert who is updating her earlier report on the same matter. Typically, persons seek to accomplish tasks as much as possible based on their prior knowledge to minimize the effort required. However, if actors are not sufficiently familiar with work tasks, hobbies or daily activities that they wish to perform, they have to learn what is necessary to realize these activities. This may happen by trial and error, by reading or by asking somebody, but all these means include learning as a component. The extent of this learning component varies from acquiring facts to restructuring one s knowledge concerning the task to be performed. After the performance, the person knows more about the object of the activity performed. In ordinary language use people do not typically speak about learning, when they accomplish work tasks or everyday activities. If a lawyer acquires information to familiarize herself with a case, or if a journalist surveys the background of an event for a feature article, they probably think that they do their job, not that they learn for accomplishing their job. Learning is embedded in the activity performed. In several learning theories human knowledge is described as knowledge structures, which consist of concepts and their relations representing some phenomenon [7]. These structures can be called, for example, mental models or schemas. If we use this definition of knowledge, then learning means gaining new or modifying and reinforcing existing knowledge structures. Naturally, one needs information inputs for changing or reinforcing current knowledge structures, that is, for learning. Typically, this implies working with information acquired to adjust it to existing or adapted knowledge structures. This means that the development of new knowledge (learning) is based on prior knowledge. We cannot pass what we know, when we seek to learn new things. The extent of learning varies from learning a fact to radically modifying one s knowledge about a phenomenon. In learning theories cognitive changes have been typically categorized as assimilation and accommodation [2]. Assimilation means addition of information into existing knowledge structures, whereas accommodation means modifying or changing existing knowledge structures. The latter refers to adding or removing concepts and their relations in the knowledge structure. The former means that the conceptual construct the knowledge structure consists of does not change, but merely new instances are added in to concepts. Some researchers have specified this distinction by distinguishing between three types of changes in knowledge structures [2]. Accretion refers to the gradual addition of factual information within existing knowledge structure without structural changes. Accretion does not change concepts and their relations in the structure, but may populate a concept with new instances or facts. Accretion is thus a synonym to assimilation. Accommodation is refined according to the degree of structural change [2]. Tuning or weak revision does not include replacing concepts or relations between concepts in the structure, but merely tuning of the scope and meaning of concepts

3 Vakkari 9 and their relations. This may include, for example, generalizing or specifying a concept. Restructuring means changing and replacing concepts and their relations in the knowledge structure. Understanding learning as conceptual change allows us to conceptualize knowledge structures as consisting of concepts and their relations. This notion matches the cognitive view of information retrieval, which conceptualizes information as knowledge structures both in humans and in documents [8, 9]. It matches also theoretical ideas concerning the dimensions of indexing and querying and their relations to retrieval effectiveness, for example, precision and recall [10], and with ideas and empirical results in the psychology of learning. This conception connects in a fertile way research traditions in information science and pedagogics. 3. Information searching There are two basic notions of information searching. The bibliographic paradigm restricts it to accessing information,that is, to querying and evaluating the search results produced by the query. The constructive, that is, learning notion extends it to include also the use of information in the documents accessed [1]. In the following, the latter is selected as the point of departure, because it represents more versatile learning. In describing the search process, it is typical to distinguish expressing information need, formulating query (selecting search terms and querying) and assessing search results (selecting sources). Belkin [11] and Ingwersen [9] sought in the 1990s to extend information searching to also include interacting with texts retrieved, that is, information use. Interaction with texts changes users understanding, and consequently the information need reflected in the search formulation. Currently, the use of information accessed is also more commonly included in the representation of the search process [12]. The latter can be divided into interacting with sources or information and synthesizing and presenting (Figure 1). Expressing the information need for search formulation and executing the query is the starting point of a search process [13]. Search formulation includes the articulation of search terms and the use of search operators if there are any. It also includes the search tactics used for reaching the search goal. The choice of search terms is a reflection of the searcher s understanding of her information need, what is insufficient in her knowledge structure. In practice, a searcher expresses her information need by selecting search terms and search tactics, that is, by formulating and reformulating queries. The learning process required for selecting terms and tactics depends at least on the searcher s level of knowledge on the search topic and system, and it is influenced also by the information provided in the search results [14] and the support tools incorporated in the search interface [15, 16]. The searcher probably reflects and reforms her understanding of the topic, and consequently on terms for querying. In selecting sources the searcher explores the document surrogates in the result list or full documents to assess the value (relevance, utility) of the documents found and to satisfy her information need, that is, to reduce the insufficiency of her knowledge. In particular, various document information elements are used for inferring the value of a source [17]. The aim of this interaction is to select sources, which probably contribute to the activity triggering the search so that they are available for immediate or later use. Interacting with sources is the phase when the searcher familiarizes herself with the sources to realize the task for which they were selected. It may include reading, making notes, extracting pieces of information, combining and organizing information and outlining [1]. The aim of this activity is to analyse, categorize and structure information in documents for preparing the final outcome of the task. The borderline to synthetizing and presenting information for task outcome is blurred, because it is impossible to determine exactly when, for example, combining and outlining information by writing for the task outcome becomes synthetizing or presenting. It is likely that, in a complex task that the actor is not familiar with, the characteristics of interacting with sources can be equated in Kuhltau s ISP model with the stages prior to information collection [1]. In any case, synthetizing and presenting include preparing the final outcome of the task. The form of the outcome depends naturally on the nature of the task. It can be a financing decision, a newspaper article or a greater understanding of some societal issue. Figure 1. Information search process.

4 Vakkari Learning in the search process After explicating the notions of both oearning and information searching, it is possible to analyse when and how learning occurs in the search process. The aim is to analyse learning at various stages of the search process. As mentioned, the extent of learning varies from accommodation to assimilation, that is, from learning facts to the radical revision of knowledge structures [2]. The point of departure in this analysis is the revision of knowledge structures, which means that a person is not familiar with the topic the task generating searching deals with. We may also say that she has an anomalous state of knowledge concerning the task [18]. Accommodation is used as an example, because in the course of task performance the modified knowledge structure stabilizes and learning changes from revision to tuning and assimilation. This kind of learning is gradual and requires time to acquire and process information for accomplishing the task. Therefore information is sought over several search sessions. This process resembles Kuhlthau s ISP model [1]. The notion of exploratory search can also be applied here, because the aim of searching is supporting learning or investigative activities [12]. In the beginning of the search a person seeks to articulate her information need into search terms. Her understanding of the topic is vague. Her knowledge structure is insufficient for expressing exactly what she wants to know. It lacks sufficient concepts and their relations for representing the topic in order to proceed in the task. Therefore, she is searching for how to structure the topic, actually conceptual structures [19, 20]. In this kind of situation there are seldom readymade conceptual structures available, but the person has to create them incrementally. There is typically not a single right solution, but many options available for how to proceed and how to specify the structure. It is likely, that neither during the initial search formulation stage nor during the initial search session, a person s conceptual structure changes radically [21]. This would require exploration of the sources selected and work with the information found. The process of making sense of the search results found is typically so demanding that most of the searcher s intellectual capacity goes to identifying sources that seem useful, leaving hardly any capacity for revealing their contribution. It is likely that in the search formulation stage the person seeks to explore the conceptual space in documents related to her own conceptual structure of the topic. This usually starts by articulating first her own conceptual understanding as a query, and then extending it by additional concepts or specifying it by reformulating the query. The exploration of the conceptual space often occurs by using vary tactics, that is, replacing existing concepts with new ones in a query [14, 22]. One may be interested, for example, in the relations of searching and learning, and explore this conceptual space first by adding to the query searching, learning new concepts like support tools and then replacing this new concept by concept evaluation. Exploring by varying tactics may reveal new concepts and conceptual relations that facilitate restructuring the searcher s knowledge structure. New concepts are potentially found when the person explores document surrogates in the result list and full documents [23]. Titles, abstracts and text passages browsed include possible ideas for restructuring her conceptual understanding of the topic. Restructuring refers to changes in concepts and in relations between concepts, in particular. An essential condition for restructuring is changes in relations between concepts. Meaning typically depends on the relations between concepts [7]. Although concepts may be identical, that is, the structure is conceptually similar, differences in their relations, that is, in factual similarity, produce differences in the meaning [24]. Relating existing and new concepts in a new way, that is, restructuring one s knowledge on a topic, is mostly a laborious and time-consuming process, which probably does not occur during search sessions, although there may be sudden insights in front of search results. Therefore, it is more typical to recognize new concepts, that is, search terms, from search results than conceptual relations. These terms are used for reformulating queries. Studies also show that during learning tasks the number and specificity of search terms used tend to increase across the search sessions [21]. Growth of knowledge means growth in the number and specificity of concepts and their interrelations. When a person has revised her conceptual structure and formulated a focus for her topic according to the ISP model [1], it will include the major concepts and their relations, but probably require tuning to better fit with the aim of her task. Tuning typically includes limiting or broadening the scope of the concepts in the knowledge structure or its applicability area. In search formulation this means using either more specific or more general search terms to present the concepts in queries. It is likely that, in tuning, factual information is also sought to support the arguments developed. Tuning leads to assimilation, that is, to the population of concepts with new instances or facts. When the conceptual structure is stable, synonyms are typically used in query formulation to represent concepts. In source selection the searcher s behaviour probably changes by changing learning. When revising knowledge structure, a person s conceptual construct about the topic is vague. Therefore, she has difficulty in recognizing useful sources, leading to loose relevance criteria [1, 25 27]. Many of the sources may be useful, but she is not able to tell which ones, because she has as yet no firm ideas on how to shape her topic. In this phase, in addition to useful sources, it is typical to

5 Vakkari 11 Table 1. Characteristics of search process by the change in knowledge structures Search stage Modification of knowledge structures Restructuring Tuning Assimilation Search formulation Few general terms Increase in the number and specificity of terms Increase in number and specificity of terms Many new terms from results Increase in the number of terms Fewer new terms from results with associative relations (facets) Varying tactics Less reformulation Less reformulation Much reformulation A few synonyms Synonyms Synonyms Long search sessions Shorter search sessions Short search sessions Source selection Vague relevance criteria Clearer relevance criteria Clear relevance criteria Number of result pages viewed is large Number of result pages viewed decreases Number of result pages viewed is small Number of sources selected is large Number of sources selected decreases Number of sources selected is small Share of selected sources of consulted sources is large Share of selected sources of consulted sources decreases Share of selected sources of consulted sources is small Share of probably relevant sources great of all sources selected Share of probably relevant sources decreases, and that of relevant sources increases Share of probably relevant sources is small, and that of relevant sources is large Interacting with sources Proportion of general background information and theoretical information is large Most time is used for assessing probably relevant sources A small share of consulted sources is used in outcome Background and conceptual information are utilized from sources Notes are taken on themes and ideas Ideas are related and combined for a focus Proportion of procedural, specific and factual information increases Increasing share of consulted sources used in outcome Procedural and specific information utilized from sources Identification of information to support and refine focus Proportion of specific and factual information increases A large share of consulted sources is used in outcome Factual and specific information utilized from sources Refining the output with factual information Rechecking sources for information initially overlooked select also sources that may be of value, but the value is not known yet. This becomes known when the sources have been explored more in detail. When tuning a knowledge structure, a person s conceptual structure is stabilized, and she is able to select documents, applying strict relevance criteria [1, 27]. She seeks to select highly useful (pertinent) documents, which seem to make a clear contribution to her task. It is likely that the absolute number of documents selected is lower in tuning compared with restructuring, but the proportion of documents used out of documents selected is greater in tuning [28]. This trend is strengthened when moving from tuning to assimilation. It is likely that the type of information used also varies by the types of learning. In restructuring knowledge, general background information, definitions of concepts and models and theoretical ideas concerning the potential topic benefit users [29]. These information types will help to form an overview of the topic and provide various conceptualizations and categorizations of it. This will help in creating and revising the user s own conceptual structure of the topic. It is likely that in restructuring people also need procedural and method information on how to proceed in the knowledge construction process [29]. Facts and specific information are likely to be more useful when one has moved from restructuring to tuning the conceptual structure, and further to assimilation. These can be applied to support arguments related to the modified structure [29]. Interacting with sources is the most crucial phase for leaning [17, 30]. Users familiarize themselves with the sources selected first with the aim of revising their understanding concerning the topic. Some of the sources are weeded out as not contributing. The rest are read for making notes of the central themes and concepts, for summarizing major points and for comparing and combining central concepts and themes. Gradually users compare and relate concepts in sources with their own conceptual understanding for restructuring their knowledge of the topic. After restructuring they are able to tune the structure and support it by factual information, and finally to create a presentation of the outcome.

6 Vakkari 12 Table 2. Criteria of leaning across sessions for restructuring knowledge structures Search stage Search formulation Source selection Interaction with sources Criteria of learning (and search success) Increase in number and specificity of terms Increase in number of terms with associative relations (facets) Increase in number of synonyms Decrease in number of reformulated queries Decrease in variability of tactics Decreased time use per search sessions Increased clarity in relevance criteria = increased ability to distinguish between relevant and nonrelevant sources Decrease in number of sources viewed in result list The proportion of sources selected of sources viewed decreases (greater decrease in precision, increase in CG) The number of sources selected decreases The share of probably relevant sources decreases, and that of relevant sources increases for all sources selected The proportion of general background information and theoretical information decreases The proportion of specific and factual information increases The proportion of procedural information is inversely U-shaped in the search process Average time used for assessing a source decreases Increasing share of sources viewed and selected used in outcome Increase in number and specificity of concepts and their interrelations in the knowledge structure Decrease in the proportion of general background and theoretical information from sources used for outcome Increase in the proportion of specific and factual information utilized from sources for outcome The proportion of procedural information used for outcome is inversely U-shaped in the learning process Based on the argumentation above, Table 1 suggests how various types of learning are associated with the stages in the search process. During the learning process, search formulations change from simple queries including a few general terms to queries consisting of several, more specific terms and also synonyms. When actors learn, their ability to structure the topic increases, leading to fewer reformulations and varying tactics, and queries with more facets of the topic [2, 14, 21, 31]. During this process, actors relevance criteria change from vague to clear, reducing the time, effort and the number of items inspected in result pages. Among the sources selected, the proportion of pertinent documents increases and that of possible relevant documents decreases [17, 25 27]. Also the type of information needed and used changes during the information search process in task performance [29, 32]. An increasing share of consulted sources is used in the outcome [28, 30]. Table 2 suggests how changes in the various stages of searching can be used as criteria of learning during the search process. These criteria are inferred from the information in Table Learning in search tasks In the previous analysis, the point of departure for searching was learning for a task that requires restructuring of one s knowledge. However, learning may occur also during search tasks. The aim of search tasks is to find and identify information or information sources, not to create new knowledge based on the sources found. These types of search tasks are typical in interactive information retrieval (IR) experiments. In the following I discuss learning in search tasks. Search tasks consist of search formulation and source selection, because they do not require interaction with sources to produce new knowledge. Learning may occur in recognizing new search terms, discarding bad ones and perhaps identifying new aspects of the search topic. Typically, it is expected that search tasks are accomplished in a single search session. Also a search topic is typically given. Therefore, it is likely that learning occurs in the form of tuning knowledge structures and identifying facts. This may include broadening or narrowing search terms or identifying new ones. It is also possible that inspecting the result list and document surrogates increases searchers understanding of the scope of the topic, increasing their ability to distinguish between relevant and non-relevant sources. Krathwohl s [33] revised Bloom s taxonomy of learning objectives can be used for characterizing learning requirements in search tasks [cf. 34, 35]. The taxonomy describes what students are expected to learn as a result of instruction.

7 Vakkari 13 The taxonomy consists of two dimensions: knowledge dimension and cognitive process dimension. The knowledge dimension includes (a) factual knowledge (knowledge of terminology and of specific details and elements), (b) conceptual knowledge (of interrelationships between elements within a larger structure), (c) procedural knowledge (how to do something) and (d) metacognitive knowledge. The cognitive process dimension consists of six categories: (a) remember (retrieving, recognizing and recalling relevant knowledge from memory); (b) understand (determining the meaning of messages through interpreting, exemplifying, classifying, summarizing etc.); (c) apply (carrying out a procedure in a given situation); (d) analyse (breaking material into its constituent parts and detecting how the parts relate to one another and to overall structure); (e) evaluate (making judgments based on criteria); and (f) create (putting elements together to form a novel, coherent whole). The complexity of cognitive processes increases along the process dimension [33]. It is expected that remembering is less complex than applying, which is less complex than creating. It is possible to apply the knowledge dimension of taxonomy to typical search tasks, which require querying to identify topically relevant information or sources. They consist of search formulation and result evaluation (source selection). It is evident that searchers need factual knowledge (knowledge of terminology and of specific details and elements) in selecting search terms and identifying relevant sources from result lists. They also apply procedural knowledge to realize the search process. Cognitive processes required for this kind of search task mainly consist of remembering, that is, recognizing search terms from search task and search results, and recalling them from one s own memory, and also recognizing relevant items from the result list. The latter may also include some understanding in determining the meaning, that is, the relevance of a particular item, or identifying various aspects of the topic. Bloom s revised taxonomy has also been used in a few studies on interactive information retrieval to design search tasks and analyse search interactions. Jansen and his colleagues [34] inferred six types of search tasks from the cognitive process dimension of the taxonomy. The tasks represented each of the learning categories in the taxonomy. They studied whether these search tasks differentiate the characteristics of Web searching, for example, the numbers of queries, unique terms and result pages viewed. The findings seemed to show that the relation between the cognitive complexity of tasks and the quantitative value of search indicators used resembled an inverted curve. The realization of search tasks at the middle level of cognitive learning required more effort in querying and result exploration compared with lower and higher levels of learning. The results are very tentative, because the validity of some search task types may be questionable owing to operationalization. Wu and colleagues [36] used the cognitive process dimension in Bloom s taxonomy [33] for creating search tasks. Their tasks included five of the cognitive learning categories except apply. The results of an experimental study indicated that search interaction increased as the level of cognitive learning increased. The indicators of interaction consisted of time on task, the number of queries issued, the number of results clicked and the number of URLs visited. 6. Types of research on searching as learning Next are presented different types of studies on relations between searching and learning by observing dependent and independent variables in the studies. In the first group of studies, various aspects of learning are treated as dependent variables, while IR process variables are considered as independent variables. In the second group, the characteristics of learning are treated as independent variables and elements in the IR process as dependent variables. The third group consists of studies characterizing search outcome, typically text, as an indicator of learning. Most of the studies that deal with learning in the search process do not conceptualize the phenomenon as learning. They observe changes in the search process, which can be interpreted as learning From searching to learning The most common type of study includes search formulation or the use of a search support tool as an independent factor and source selection as a dependent factor. The typical aim of this study type is to find out how search formulation or result list inspection behaviour is associated with search effectiveness. The most common independent variable used is the indicator based on the number of relevant documents found, not learning. However, a byproduct of these studies may be results concerning identifying keywords from search results or new aspects of search topic [37]. These findings are not conceptualized as learning although they include learning processes remembering and understanding. However, in an exceptional study, Moraveji and his colleagues [15] analysed how tactical search feature tips were associated with search performance. These tips expanded user awareness of the task-relevant tools and features of the search application. The group exposed to the tips performed the search tasks in a shorter time compared with the control group. This difference

8 Vakkari 14 remained about one week later in a follow-up study with no tips shown. Also the number of correct answers was larger in the experimental group. Xie and Cool [16] have identified 15 different help-seeking situations during search sessions. There are also studies that extend the scope of analysis from search formulation and source selection to cover also either interacting with sources or the characteristics of the outcome (end product) of the search like a text. Hersh and his colleagues [38] were among the first to empirically evaluate search systems from the viewpoint of learning task performance. They assessed to what extent students were able to solve clinical problems by using two information retrieval systems. In these studies search formulation or source selection with possible search tools is used as an independent factor and the use of sources and search outcome as dependent variables. They typically seek to find out how the search process contributes to source use or to the end product of search. For example, Liu and Belkin [39] investigated whether task type in writing a feature article has an effect on search behaviour and writing task outcome over sessions. The results showed that neither task type nor task session affected outcome, but the actor s topic familiarity and task experience did have a positive effect on outcome. Irrespective of the task type, sub-chapters and the final article consisted of equal numbers of sentences and facts. The actor s search and writing behaviour affected outcome. More time proportionally used for writing instead of searching was associated with the increase in the number of sentences and facts in the report. However, a further analysis of the data showed that users perceived that their knowledge level changed after each of the three sessions [40]. Also studies by Bron and colleagues [41] and by Vakkari and Huuskonen [30] analysed how search process variables contributed to the outcome of the search. The outcome was in the former the quality of the research question created with the help of material retrieved, and in the latter the proportion of useful retrieved documents cited in the assigned essay and the quality of the essay. In both studies, the quality of outcome was assessed by experts. Diverging from the first study type, it is possible to conceptualize the evaluation of information resources found as a process of comprehension and learning, when one is studying how, for example, various interfaces are related to the evaluation of information resources retrieved. Thus, the dependent factor used does not represent search effectiveness, but the characteristics of cognitive processing in evaluating search results. Butcher and colleagues [42] studied how preservice teachers search educational material for classroom instructions using two different interfaces. The three search tasks in the study emulated online, daily instructional planning activities. The participants were asked to find digital materials for learners of various types (a) on the water cycle, (b) on cell biology and (c) to visualize how plate interactions relate to natural phenomena. Thus, the search tasks requested topical visual material. The study compared between the interfaces the cognitive processes in which participants engaged while they conducted searches and analysed lists of returned materials. The major data were search logs and participants think-alouds during the search process. The utterances concerning the evaluation of resources were, for example, classified into three categories that were informed by the comprehension integration model of comprehension [7]. The definition of these categories resembled some of those in Bloom s taxonomy. The categories reflected the information processing depth shown in evaluation. Also, other measures concerning the processing of information in utterances were used. The results showed that an interface representing the conceptual structure of a field supports deeper cognitive processing and evaluation of results during information search compared with an established keyword search system. The interface led actors to spend more time exploring domain ideas and relationships and less time exploring individual resources. This example shows that learning can be studied using a familiar research design in interactive IR with typical search tasks, if the search process is conceptualized and operationalized as learning or cognitive processes-like result evaluation in this case. Thus, if we wish to study information searching as learning, we have to conceptualize searching as learning and cognitive processes and use respective factors as dependent variables From learning to searching Some studies have explored how search formulation and source selection vary by the level (complexity) of cognitive learning processes based on Bloom s taxonomy [34 36]. There are also studies that categorize search tasks using characteristics that can be reduced at least to a certain extent to Bloom s taxonomy. For example, Liu et al. [40] characterized search tasks based on typical journalistic tasks like background information collection or copyediting. The type of search task was used as independent variables and search interactions as dependent ones. In these studies the complexity of learning processes is used as an independent (comparative) variable, and search interactions and the use of search support tool as dependent variables. To generalize, in these studies characteristics of learning like the complexity of learning or the type of knowledge required in learning are used as independent variables and search interactions as dependent ones. There is also a stream of studies mainly based on Kuhlthau s work [1], which explain the variation of search process variables by the stage in the ISP model. They treat learning as an independent variable and search interactions as

9 Vakkari 15 dependent variables. These studies suppose that participants gain and modify their knowledge of the task topic, that is, learn when they proceed in the ISP model. For example, Vakkari [14, 21] has demonstrated in longitudinal studies that users term selection, search tactics, relevance assessments and the type of information used vary systematically by the stages in ISP model, that is, by progressing learning. Also Wildemuth [43] has used in a longitudinal study participants progress in learning, that is, the growth of their domain knowledge, as an independent variable, and search tactics as dependent variables. The results show that search tactics changed over time when participants domain knowledge changed Search outcome as an indicator of learning An increasing interest in the outcome of searching has led to the development of measures of learning inferred from the outcome. A typical outcome analysed is a written text based on search results. Wilson and Wilson [44] used Bloom s revised taxonomy to create measures for the depth of learning in written summary texts based on documents retrieved. The first measure is the quality of facts recalled in the text. It reflects Bloom s category understanding. The second measure is the interpretation of facts into statements. It reflects to what extent facts were combined in statements to draw conclusions. This measure is based on Bloom s category analysis. The third measure reflects Bloom s category evaluation, by identifying statements that compared facts or used facts to raise questions about other statements. Thus, these measures indicate the depth of learning shown in the three levels of learning in Bloom s taxonomy. Sormunen and colleagues [45] developed a method for analysing information use in source-based writing. By information use is meant text transformations from sources of texts composed. The method identifies how the text from source documents is modified and composed to a new text. The text transformation variables are (a) the degree of paraphrasing, (b) the degree of synthesis, (c) credibility on building arguments, (d) accuracy of citing, (e) statement type and (f) source type. These variables could to some extent be translated to represent Bloom s cognitive processing categories. For example, the categories in the degree of paraphrasing stretch from copy paste and paraphrasing to own text. Copy paste is equal to remembering, paraphrasing to understanding and own text to analysing and evaluating, in particular. Some studies in this genre, like Saito and colleagues [46] have analysed how participants conceptual structures changed when they searched documents for writing magazine articles on certain topics. The participants were asked to draw a conceptual map on the topic of search before and after a 15 min search. From the concept maps the number of concept words (nodes), links between nodes and link words were identified. This approach seeks explicitly to uncover changes in participants conceptual structure, that is, learning. It also facilitates identification of the degree of structural change in knowledge, that is, to what extent the structure is reformulated or tuned. The studies presented so far are studies on interactive information retrieval. However, there is at least one query log study that analyses learning while searching. Eickhoff [47] investigated within session and cross-session developments of expertise. The study focused on how users query terms and search behaviour on a topic changed over time. They inferred from previous studies characteristics of learning while searching. They build indicators of learning using data from queries and results visited. The indicators of learning included, for example, focus, that is, the extent of the topical space explored by users, the display time of items viewed and query complexity as measured by a reading-level metrics. The results showed, for example, that the number of domains in a search result page increased as well as the complexity of keywords over the course of a session. The results suggest that visits in documents on result lists contributed to learning, that is, by inferring information that produced queries with more complex keywords and consequently results with increased focus. 7. Conclusions There is a growing trend in studies to expand and re-conceptualize the search process from querying and inspecting search results to cover also working with search results for creating the outcome of a search. This implies that, along with traditional independent variables, that is, relevance metrics, variables indicating the characteristics of source use and search outcome should be applied. This shift signifies understanding search not primarily as only to gather and collect information, but as a series of tasks within a constructive process, like Kuhlthau suggested [1]. Actors need both support in accessing sources and guidance in the construction process, that is, in learning. The early advocates of this view were also Belkin [11] and Marchionini [13]. A consequence of this trend has been the change in the role of relevance metrics from independent variable to process variable between search formulation and source use. In addition to extending the search process towards source use and search outcome, it is essential to conceptualize term selection and relevance assessments as thinking and learning processes, not just count the number of terms or relevant items. Knowing better the reasoning processes behind term selection and relevance assessments probably helps in

10 Vakkari 16 understanding searching as learning. There are studies that have sought to analyse cognitive processes associated with term selection and relevance assessments [17, 23, 27, 48]. The task-based notion of information searching has been perhaps the major driving force for expanding the framework of information searching [3]. Although it has produced some interesting empirical results [30, 39, 49], it seems to lack a coherent conceptual framework on which to build empirical research. One of the major conceptualizations in task-based studies has been Kuhlthau s ISP model, which is deeply rooted in pedagogy [1]. It actually models searching as learning. It hints that conceptualizing searching as a learning process may provide our research with richer tools for analysing how humans act and make sense of information accessed and encountered in various daily activities. Learning occurs in those activities when humans do not have sufficient information to accomplish them. It is a matter of learning, although in ordinary language use people do not typically speak about learning in this context. When exploring studies that use interaction with sources retrieved and task outcome as dependent variables, the nature of the findings was astonishing. They were surprising, but in some sense evident. Results seem to show that the more time and effort users spent at the end phases of the whole search process, the better the quality of the search outcome was. The more users were able to focus on results inspection instead of querying and on working with information in documents found instead of result inspection, the better this seemed to be to the process of construction for proceeding in the task [cf. 30, 39, 42]. Also a recent query log study [47] seems to hint at this. Thus, it is not enough to provide users with a high-quality result list; they also need tools that help in making sense of, structuring and manipulating search results and sources for restructuring and tuning their knowledge structures, that is, processing the expected outcome of the search. This matches Kuhlthau s idea of providing support both in accessing sources and in construction process [1]. These findings provide system design with new challenges. It would be important to create tools that support, for example, structuring the result list according to the potential conceptual structures in documents, and restructuring the list by clicking desired concepts to combine them. This would help users to explore the conceptual space in documents in relation to their insufficient conceptual structures. Also the type of information in documents (e.g. general background information, concepts, procedural information, facts, etc.) would be a useful categorization of search results when users proceed through their task according to Kuhltau s model [1, 29]. There is also an urgent need to develop tools that support users to interact with sources for constructing an end-product by structuring and manipulating information in those sources. There are already attempts in educational psychology to develop and evaluate cognitive personalization technologies to support students sense-making [50]. Conceptualizing searching as learning, and designing new tools to support learning while searching, evidently requires developing new evaluation indicators. Evaluation begins by articulating the goal of the object of evaluation. The goal provides ideas for inferring indicators and criteria of success in the use of the tool observed. Bloom s taxonomy of learning objectives [33] can be used for characterizing learning requirements in using search tools and also in designing search tasks [cf. 34, 35]. The taxonomy describes what students are expected to learn as a result of instruction. This can be extended to what users are expected to learn as a result of using support tools provided. Therefore, the taxonomy is a good point of departure in designing search tasks and evaluation goals and criteria. Conceptualizing learning as conceptual change connects research on information searching to both a cognitive view of information searching and major theoretical ideas in information retrieval, and also theoretical ideas and empirical results in the psychology of learning. This provides a fertile point of departure for those wishing to study searching as learning. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. References [1] Kuhlthau C. Seeking meaning. Norwood, NJ: Ablex, [2] Zhang P and Soergel D. Towards a comprehensive model of the cognitive process and mechanisms of individual sensemaking. Journal of the Association for Information Science and Technology 2014; 65(9): [3] Vakkari P. Task based information searching. In: Annual review of information science and technology 37. Medford, NJ.: Information Today, [4] Blandford A and Attfield S. Interacting with Information. San Rafael, CA: Morgan and Claypool, [5] Schacter DL, Gilbert DT and Wegner DM. Psychology, 2nd edn. New York: Worth, [6] Ormrod J. Human learning, 6th edn. Boston, MA: Pearson, [7] Kintsch W. Comprehension. A paradigm for cognition. Cambridge University Press: Cambridge, 1998.

10.2. Behavior models

10.2. Behavior models User behavior research 10.2. Behavior models Overview Why do users seek information? How do they seek information? How do they search for information? How do they use libraries? These questions are addressed

More information

Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge

Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Innov High Educ (2009) 34:93 103 DOI 10.1007/s10755-009-9095-2 Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Phyllis Blumberg Published online: 3 February

More information

PEDAGOGICAL LEARNING WALKS: MAKING THE THEORY; PRACTICE

PEDAGOGICAL LEARNING WALKS: MAKING THE THEORY; PRACTICE PEDAGOGICAL LEARNING WALKS: MAKING THE THEORY; PRACTICE DR. BEV FREEDMAN B. Freedman OISE/Norway 2015 LEARNING LEADERS ARE Discuss and share.. THE PURPOSEFUL OF CLASSROOM/SCHOOL OBSERVATIONS IS TO OBSERVE

More information

Developing True/False Test Sheet Generating System with Diagnosing Basic Cognitive Ability

Developing True/False Test Sheet Generating System with Diagnosing Basic Cognitive Ability Developing True/False Test Sheet Generating System with Diagnosing Basic Cognitive Ability Shih-Bin Chen Dept. of Information and Computer Engineering, Chung-Yuan Christian University Chung-Li, Taiwan

More information

Abstractions and the Brain

Abstractions and the Brain Abstractions and the Brain Brian D. Josephson Department of Physics, University of Cambridge Cavendish Lab. Madingley Road Cambridge, UK. CB3 OHE bdj10@cam.ac.uk http://www.tcm.phy.cam.ac.uk/~bdj10 ABSTRACT

More information

CEFR Overall Illustrative English Proficiency Scales

CEFR Overall Illustrative English Proficiency Scales CEFR Overall Illustrative English Proficiency s CEFR CEFR OVERALL ORAL PRODUCTION Has a good command of idiomatic expressions and colloquialisms with awareness of connotative levels of meaning. Can convey

More information

Metadiscourse in Knowledge Building: A question about written or verbal metadiscourse

Metadiscourse in Knowledge Building: A question about written or verbal metadiscourse Metadiscourse in Knowledge Building: A question about written or verbal metadiscourse Rolf K. Baltzersen Paper submitted to the Knowledge Building Summer Institute 2013 in Puebla, Mexico Author: Rolf K.

More information

Activities, Exercises, Assignments Copyright 2009 Cem Kaner 1

Activities, Exercises, Assignments Copyright 2009 Cem Kaner 1 Patterns of activities, iti exercises and assignments Workshop on Teaching Software Testing January 31, 2009 Cem Kaner, J.D., Ph.D. kaner@kaner.com Professor of Software Engineering Florida Institute of

More information

Ph.D. in Behavior Analysis Ph.d. i atferdsanalyse

Ph.D. in Behavior Analysis Ph.d. i atferdsanalyse Program Description Ph.D. in Behavior Analysis Ph.d. i atferdsanalyse 180 ECTS credits Approval Approved by the Norwegian Agency for Quality Assurance in Education (NOKUT) on the 23rd April 2010 Approved

More information

Mathematics Program Assessment Plan

Mathematics Program Assessment Plan Mathematics Program Assessment Plan Introduction This assessment plan is tentative and will continue to be refined as needed to best fit the requirements of the Board of Regent s and UAS Program Review

More information

Analysis: Evaluation: Knowledge: Comprehension: Synthesis: Application:

Analysis: Evaluation: Knowledge: Comprehension: Synthesis: Application: In 1956, Benjamin Bloom headed a group of educational psychologists who developed a classification of levels of intellectual behavior important in learning. Bloom found that over 95 % of the test questions

More information

A Note on Structuring Employability Skills for Accounting Students

A Note on Structuring Employability Skills for Accounting Students A Note on Structuring Employability Skills for Accounting Students Jon Warwick and Anna Howard School of Business, London South Bank University Correspondence Address Jon Warwick, School of Business, London

More information

What is PDE? Research Report. Paul Nichols

What is PDE? Research Report. Paul Nichols What is PDE? Research Report Paul Nichols December 2013 WHAT IS PDE? 1 About Pearson Everything we do at Pearson grows out of a clear mission: to help people make progress in their lives through personalized

More information

Protocol for using the Classroom Walkthrough Observation Instrument

Protocol for using the Classroom Walkthrough Observation Instrument Protocol for using the Classroom Walkthrough Observation Instrument Purpose: The purpose of this instrument is to document technology integration in classrooms. Information is recorded about teaching style

More information

Unit 3. Design Activity. Overview. Purpose. Profile

Unit 3. Design Activity. Overview. Purpose. Profile Unit 3 Design Activity Overview Purpose The purpose of the Design Activity unit is to provide students with experience designing a communications product. Students will develop capability with the design

More information

Developing Students Research Proposal Design through Group Investigation Method

Developing Students Research Proposal Design through Group Investigation Method IOSR Journal of Research & Method in Education (IOSR-JRME) e-issn: 2320 7388,p-ISSN: 2320 737X Volume 7, Issue 1 Ver. III (Jan. - Feb. 2017), PP 37-43 www.iosrjournals.org Developing Students Research

More information

Introduction. 1. Evidence-informed teaching Prelude

Introduction. 1. Evidence-informed teaching Prelude 1. Evidence-informed teaching 1.1. Prelude A conversation between three teachers during lunch break Rik: Barbara: Rik: Cristina: Barbara: Rik: Cristina: Barbara: Rik: Barbara: Cristina: Why is it that

More information

AGENDA LEARNING THEORIES LEARNING THEORIES. Advanced Learning Theories 2/22/2016

AGENDA LEARNING THEORIES LEARNING THEORIES. Advanced Learning Theories 2/22/2016 AGENDA Advanced Learning Theories Alejandra J. Magana, Ph.D. admagana@purdue.edu Introduction to Learning Theories Role of Learning Theories and Frameworks Learning Design Research Design Dual Coding Theory

More information

Entrepreneurial Discovery and the Demmert/Klein Experiment: Additional Evidence from Germany

Entrepreneurial Discovery and the Demmert/Klein Experiment: Additional Evidence from Germany Entrepreneurial Discovery and the Demmert/Klein Experiment: Additional Evidence from Germany Jana Kitzmann and Dirk Schiereck, Endowed Chair for Banking and Finance, EUROPEAN BUSINESS SCHOOL, International

More information

Loughton School s curriculum evening. 28 th February 2017

Loughton School s curriculum evening. 28 th February 2017 Loughton School s curriculum evening 28 th February 2017 Aims of this session Share our approach to teaching writing, reading, SPaG and maths. Share resources, ideas and strategies to support children's

More information

Stimulating Techniques in Micro Teaching. Puan Ng Swee Teng Ketua Program Kursus Lanjutan U48 Kolej Sains Kesihatan Bersekutu, SAS, Ulu Kinta

Stimulating Techniques in Micro Teaching. Puan Ng Swee Teng Ketua Program Kursus Lanjutan U48 Kolej Sains Kesihatan Bersekutu, SAS, Ulu Kinta Stimulating Techniques in Micro Teaching Puan Ng Swee Teng Ketua Program Kursus Lanjutan U48 Kolej Sains Kesihatan Bersekutu, SAS, Ulu Kinta Learning Objectives General Objectives: At the end of the 2

More information

EQuIP Review Feedback

EQuIP Review Feedback EQuIP Review Feedback Lesson/Unit Name: On the Rainy River and The Red Convertible (Module 4, Unit 1) Content Area: English language arts Grade Level: 11 Dimension I Alignment to the Depth of the CCSS

More information

USER ADAPTATION IN E-LEARNING ENVIRONMENTS

USER ADAPTATION IN E-LEARNING ENVIRONMENTS USER ADAPTATION IN E-LEARNING ENVIRONMENTS Paraskevi Tzouveli Image, Video and Multimedia Systems Laboratory School of Electrical and Computer Engineering National Technical University of Athens tpar@image.

More information

VIEW: An Assessment of Problem Solving Style

VIEW: An Assessment of Problem Solving Style 1 VIEW: An Assessment of Problem Solving Style Edwin C. Selby, Donald J. Treffinger, Scott G. Isaksen, and Kenneth Lauer This document is a working paper, the purposes of which are to describe the three

More information

Document number: 2013/ Programs Committee 6/2014 (July) Agenda Item 42.0 Bachelor of Engineering with Honours in Software Engineering

Document number: 2013/ Programs Committee 6/2014 (July) Agenda Item 42.0 Bachelor of Engineering with Honours in Software Engineering Document number: 2013/0006139 Programs Committee 6/2014 (July) Agenda Item 42.0 Bachelor of Engineering with Honours in Software Engineering Program Learning Outcomes Threshold Learning Outcomes for Engineering

More information

What is Thinking (Cognition)?

What is Thinking (Cognition)? What is Thinking (Cognition)? Edward De Bono says that thinking is... the deliberate exploration of experience for a purpose. The action of thinking is an exploration, so when one thinks one investigates,

More information

Classroom Assessment Techniques (CATs; Angelo & Cross, 1993)

Classroom Assessment Techniques (CATs; Angelo & Cross, 1993) Classroom Assessment Techniques (CATs; Angelo & Cross, 1993) From: http://warrington.ufl.edu/itsp/docs/instructor/assessmenttechniques.pdf Assessing Prior Knowledge, Recall, and Understanding 1. Background

More information

Think A F R I C A when assessing speaking. C.E.F.R. Oral Assessment Criteria. Think A F R I C A - 1 -

Think A F R I C A when assessing speaking. C.E.F.R. Oral Assessment Criteria. Think A F R I C A - 1 - C.E.F.R. Oral Assessment Criteria Think A F R I C A - 1 - 1. The extracts in the left hand column are taken from the official descriptors of the CEFR levels. How would you grade them on a scale of low,

More information

Scoring Guide for Candidates For retake candidates who began the Certification process in and earlier.

Scoring Guide for Candidates For retake candidates who began the Certification process in and earlier. Adolescence and Young Adulthood SOCIAL STUDIES HISTORY For retake candidates who began the Certification process in 2013-14 and earlier. Part 1 provides you with the tools to understand and interpret your

More information

Concept mapping instrumental support for problem solving

Concept mapping instrumental support for problem solving 40 Int. J. Cont. Engineering Education and Lifelong Learning, Vol. 18, No. 1, 2008 Concept mapping instrumental support for problem solving Slavi Stoyanov* Open University of the Netherlands, OTEC, P.O.

More information

This Performance Standards include four major components. They are

This Performance Standards include four major components. They are Environmental Physics Standards The Georgia Performance Standards are designed to provide students with the knowledge and skills for proficiency in science. The Project 2061 s Benchmarks for Science Literacy

More information

Rule Learning With Negation: Issues Regarding Effectiveness

Rule Learning With Negation: Issues Regarding Effectiveness Rule Learning With Negation: Issues Regarding Effectiveness S. Chua, F. Coenen, G. Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX Liverpool, United

More information

Observing Teachers: The Mathematics Pedagogy of Quebec Francophone and Anglophone Teachers

Observing Teachers: The Mathematics Pedagogy of Quebec Francophone and Anglophone Teachers Observing Teachers: The Mathematics Pedagogy of Quebec Francophone and Anglophone Teachers Dominic Manuel, McGill University, Canada Annie Savard, McGill University, Canada David Reid, Acadia University,

More information

Full text of O L O W Science As Inquiry conference. Science as Inquiry

Full text of O L O W Science As Inquiry conference. Science as Inquiry Page 1 of 5 Full text of O L O W Science As Inquiry conference Reception Meeting Room Resources Oceanside Unifying Concepts and Processes Science As Inquiry Physical Science Life Science Earth & Space

More information

Motivation to e-learn within organizational settings: What is it and how could it be measured?

Motivation to e-learn within organizational settings: What is it and how could it be measured? Motivation to e-learn within organizational settings: What is it and how could it be measured? Maria Alexandra Rentroia-Bonito and Joaquim Armando Pires Jorge Departamento de Engenharia Informática Instituto

More information

Volume 20, AND INFORMATION BEHAVIOUR THEORIES/MODELS* *this paper was partially presented as a PHD thesis (Aydin, 2015)

Volume 20, AND INFORMATION BEHAVIOUR THEORIES/MODELS* *this paper was partially presented as a PHD thesis (Aydin, 2015) Volume 20, 2017 COGNITION TO COLLABORATION: USER-CENTRIC APPROACH AND INFORMATION BEHAVIOUR THEORIES/MODELS* *this paper was partially presented as a PHD thesis (Aydin, 2015) Alperen Mehmet Aydin ABSTRACT

More information

MASTER S THESIS GUIDE MASTER S PROGRAMME IN COMMUNICATION SCIENCE

MASTER S THESIS GUIDE MASTER S PROGRAMME IN COMMUNICATION SCIENCE MASTER S THESIS GUIDE MASTER S PROGRAMME IN COMMUNICATION SCIENCE University of Amsterdam Graduate School of Communication Kloveniersburgwal 48 1012 CX Amsterdam The Netherlands E-mail address: scripties-cw-fmg@uva.nl

More information

Guidelines for Writing an Internship Report

Guidelines for Writing an Internship Report Guidelines for Writing an Internship Report Master of Commerce (MCOM) Program Bahauddin Zakariya University, Multan Table of Contents Table of Contents... 2 1. Introduction.... 3 2. The Required Components

More information

Enduring Understandings: Students will understand that

Enduring Understandings: Students will understand that ART Pop Art and Technology: Stage 1 Desired Results Established Goals TRANSFER GOAL Students will: - create a value scale using at least 4 values of grey -explain characteristics of the Pop art movement

More information

On-Line Data Analytics

On-Line Data Analytics International Journal of Computer Applications in Engineering Sciences [VOL I, ISSUE III, SEPTEMBER 2011] [ISSN: 2231-4946] On-Line Data Analytics Yugandhar Vemulapalli #, Devarapalli Raghu *, Raja Jacob

More information

AQUA: An Ontology-Driven Question Answering System

AQUA: An Ontology-Driven Question Answering System AQUA: An Ontology-Driven Question Answering System Maria Vargas-Vera, Enrico Motta and John Domingue Knowledge Media Institute (KMI) The Open University, Walton Hall, Milton Keynes, MK7 6AA, United Kingdom.

More information

Candidates must achieve a grade of at least C2 level in each examination in order to achieve the overall qualification at C2 Level.

Candidates must achieve a grade of at least C2 level in each examination in order to achieve the overall qualification at C2 Level. The Test of Interactive English, C2 Level Qualification Structure The Test of Interactive English consists of two units: Unit Name English English Each Unit is assessed via a separate examination, set,

More information

WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING AND TEACHING OF PROBLEM SOLVING

WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING AND TEACHING OF PROBLEM SOLVING From Proceedings of Physics Teacher Education Beyond 2000 International Conference, Barcelona, Spain, August 27 to September 1, 2000 WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING

More information

A cognitive perspective on pair programming

A cognitive perspective on pair programming Association for Information Systems AIS Electronic Library (AISeL) AMCIS 2006 Proceedings Americas Conference on Information Systems (AMCIS) December 2006 A cognitive perspective on pair programming Radhika

More information

Systematic reviews in theory and practice for library and information studies

Systematic reviews in theory and practice for library and information studies Systematic reviews in theory and practice for library and information studies Sue F. Phelps, Nicole Campbell Abstract This article is about the use of systematic reviews as a research methodology in library

More information

Ontologies vs. classification systems

Ontologies vs. classification systems Ontologies vs. classification systems Bodil Nistrup Madsen Copenhagen Business School Copenhagen, Denmark bnm.isv@cbs.dk Hanne Erdman Thomsen Copenhagen Business School Copenhagen, Denmark het.isv@cbs.dk

More information

A Metacognitive Approach to Support Heuristic Solution of Mathematical Problems

A Metacognitive Approach to Support Heuristic Solution of Mathematical Problems A Metacognitive Approach to Support Heuristic Solution of Mathematical Problems John TIONG Yeun Siew Centre for Research in Pedagogy and Practice, National Institute of Education, Nanyang Technological

More information

Information Literacy Competency Standards for Higher Education

Information Literacy Competency Standards for Higher Education Information Literacy Competency Standards for Higher Education Approved by the Board of Directors of the Association of College and Research Libraries on January 18, 2000 Endorsed by the American Association

More information

South Carolina English Language Arts

South Carolina English Language Arts South Carolina English Language Arts A S O F J U N E 2 0, 2 0 1 0, T H I S S TAT E H A D A D O P T E D T H E CO M M O N CO R E S TAT E S TA N DA R D S. DOCUMENTS REVIEWED South Carolina Academic Content

More information

Assessing Functional Relations: The Utility of the Standard Celeration Chart

Assessing Functional Relations: The Utility of the Standard Celeration Chart Behavioral Development Bulletin 2015 American Psychological Association 2015, Vol. 20, No. 2, 163 167 1942-0722/15/$12.00 http://dx.doi.org/10.1037/h0101308 Assessing Functional Relations: The Utility

More information

Automating Outcome Based Assessment

Automating Outcome Based Assessment Automating Outcome Based Assessment Suseel K Pallapu Graduate Student Department of Computing Studies Arizona State University Polytechnic (East) 01 480 449 3861 harryk@asu.edu ABSTRACT In the last decade,

More information

CWIS 23,3. Nikolaos Avouris Human Computer Interaction Group, University of Patras, Patras, Greece

CWIS 23,3. Nikolaos Avouris Human Computer Interaction Group, University of Patras, Patras, Greece The current issue and full text archive of this journal is available at wwwemeraldinsightcom/1065-0741htm CWIS 138 Synchronous support and monitoring in web-based educational systems Christos Fidas, Vasilios

More information

Taxonomy of the cognitive domain: An example of architectural education program

Taxonomy of the cognitive domain: An example of architectural education program Available online at www.sciencedirect.com ScienceDirect Procedia - Social and Behavioral Sciences 174 ( 2015 ) 3272 3277 INTE 2014 Taxonomy of the cognitive domain: An example of architectural education

More information

Digital Fabrication and Aunt Sarah: Enabling Quadratic Explorations via Technology. Michael L. Connell University of Houston - Downtown

Digital Fabrication and Aunt Sarah: Enabling Quadratic Explorations via Technology. Michael L. Connell University of Houston - Downtown Digital Fabrication and Aunt Sarah: Enabling Quadratic Explorations via Technology Michael L. Connell University of Houston - Downtown Sergei Abramovich State University of New York at Potsdam Introduction

More information

A Minimalist Approach to Code-Switching. In the field of linguistics, the topic of bilingualism is a broad one. There are many

A Minimalist Approach to Code-Switching. In the field of linguistics, the topic of bilingualism is a broad one. There are many Schmidt 1 Eric Schmidt Prof. Suzanne Flynn Linguistic Study of Bilingualism December 13, 2013 A Minimalist Approach to Code-Switching In the field of linguistics, the topic of bilingualism is a broad one.

More information

TU-E2090 Research Assignment in Operations Management and Services

TU-E2090 Research Assignment in Operations Management and Services Aalto University School of Science Operations and Service Management TU-E2090 Research Assignment in Operations Management and Services Version 2016-08-29 COURSE INSTRUCTOR: OFFICE HOURS: CONTACT: Saara

More information

Monitoring Metacognitive abilities in children: A comparison of children between the ages of 5 to 7 years and 8 to 11 years

Monitoring Metacognitive abilities in children: A comparison of children between the ages of 5 to 7 years and 8 to 11 years Monitoring Metacognitive abilities in children: A comparison of children between the ages of 5 to 7 years and 8 to 11 years Abstract Takang K. Tabe Department of Educational Psychology, University of Buea

More information

LISTENING STRATEGIES AWARENESS: A DIARY STUDY IN A LISTENING COMPREHENSION CLASSROOM

LISTENING STRATEGIES AWARENESS: A DIARY STUDY IN A LISTENING COMPREHENSION CLASSROOM LISTENING STRATEGIES AWARENESS: A DIARY STUDY IN A LISTENING COMPREHENSION CLASSROOM Frances L. Sinanu Victoria Usadya Palupi Antonina Anggraini S. Gita Hastuti Faculty of Language and Literature Satya

More information

Grade 4. Common Core Adoption Process. (Unpacked Standards)

Grade 4. Common Core Adoption Process. (Unpacked Standards) Grade 4 Common Core Adoption Process (Unpacked Standards) Grade 4 Reading: Literature RL.4.1 Refer to details and examples in a text when explaining what the text says explicitly and when drawing inferences

More information

Arizona s English Language Arts Standards th Grade ARIZONA DEPARTMENT OF EDUCATION HIGH ACADEMIC STANDARDS FOR STUDENTS

Arizona s English Language Arts Standards th Grade ARIZONA DEPARTMENT OF EDUCATION HIGH ACADEMIC STANDARDS FOR STUDENTS Arizona s English Language Arts Standards 11-12th Grade ARIZONA DEPARTMENT OF EDUCATION HIGH ACADEMIC STANDARDS FOR STUDENTS 11 th -12 th Grade Overview Arizona s English Language Arts Standards work together

More information

A Systems Approach to Principal and Teacher Effectiveness From Pivot Learning Partners

A Systems Approach to Principal and Teacher Effectiveness From Pivot Learning Partners A Systems Approach to Principal and Teacher Effectiveness From Pivot Learning Partners About Our Approach At Pivot Learning Partners (PLP), we help school districts build the systems, structures, and processes

More information

BENCHMARK TREND COMPARISON REPORT:

BENCHMARK TREND COMPARISON REPORT: National Survey of Student Engagement (NSSE) BENCHMARK TREND COMPARISON REPORT: CARNEGIE PEER INSTITUTIONS, 2003-2011 PREPARED BY: ANGEL A. SANCHEZ, DIRECTOR KELLI PAYNE, ADMINISTRATIVE ANALYST/ SPECIALIST

More information

Statistical Analysis of Climate Change, Renewable Energies, and Sustainability An Independent Investigation for Introduction to Statistics

Statistical Analysis of Climate Change, Renewable Energies, and Sustainability An Independent Investigation for Introduction to Statistics 5/22/2012 Statistical Analysis of Climate Change, Renewable Energies, and Sustainability An Independent Investigation for Introduction to Statistics College of Menominee Nation & University of Wisconsin

More information

ECE-492 SENIOR ADVANCED DESIGN PROJECT

ECE-492 SENIOR ADVANCED DESIGN PROJECT ECE-492 SENIOR ADVANCED DESIGN PROJECT Meeting #3 1 ECE-492 Meeting#3 Q1: Who is not on a team? Q2: Which students/teams still did not select a topic? 2 ENGINEERING DESIGN You have studied a great deal

More information

General study plan for third-cycle programmes in Sociology

General study plan for third-cycle programmes in Sociology Date of adoption: 07/06/2017 Ref. no: 2017/3223-4.1.1.2 Faculty of Social Sciences Third-cycle education at Linnaeus University is regulated by the Swedish Higher Education Act and Higher Education Ordinance

More information

A Study of Metacognitive Awareness of Non-English Majors in L2 Listening

A Study of Metacognitive Awareness of Non-English Majors in L2 Listening ISSN 1798-4769 Journal of Language Teaching and Research, Vol. 4, No. 3, pp. 504-510, May 2013 Manufactured in Finland. doi:10.4304/jltr.4.3.504-510 A Study of Metacognitive Awareness of Non-English Majors

More information

Management of time resources for learning through individual study in higher education

Management of time resources for learning through individual study in higher education Available online at www.sciencedirect.com Procedia - Social and Behavioral Scienc es 76 ( 2013 ) 13 18 5th International Conference EDU-WORLD 2012 - Education Facing Contemporary World Issues Management

More information

Formative Assessment in Mathematics. Part 3: The Learner s Role

Formative Assessment in Mathematics. Part 3: The Learner s Role Formative Assessment in Mathematics Part 3: The Learner s Role Dylan Wiliam Equals: Mathematics and Special Educational Needs 6(1) 19-22; Spring 2000 Introduction This is the last of three articles reviewing

More information

CAAP. Content Analysis Report. Sample College. Institution Code: 9011 Institution Type: 4-Year Subgroup: none Test Date: Spring 2011

CAAP. Content Analysis Report. Sample College. Institution Code: 9011 Institution Type: 4-Year Subgroup: none Test Date: Spring 2011 CAAP Content Analysis Report Institution Code: 911 Institution Type: 4-Year Normative Group: 4-year Colleges Introduction This report provides information intended to help postsecondary institutions better

More information

Timeline. Recommendations

Timeline. Recommendations Introduction Advanced Placement Course Credit Alignment Recommendations In 2007, the State of Ohio Legislature passed legislation mandating the Board of Regents to recommend and the Chancellor to adopt

More information

Critical Thinking in Everyday Life: 9 Strategies

Critical Thinking in Everyday Life: 9 Strategies Critical Thinking in Everyday Life: 9 Strategies Most of us are not what we could be. We are less. We have great capacity. But most of it is dormant; most is undeveloped. Improvement in thinking is like

More information

Copyright Corwin 2015

Copyright Corwin 2015 2 Defining Essential Learnings How do I find clarity in a sea of standards? For students truly to be able to take responsibility for their learning, both teacher and students need to be very clear about

More information

Assessment System for M.S. in Health Professions Education (rev. 4/2011)

Assessment System for M.S. in Health Professions Education (rev. 4/2011) Assessment System for M.S. in Health Professions Education (rev. 4/2011) Health professions education programs - Conceptual framework The University of Rochester interdisciplinary program in Health Professions

More information

THE WEB 2.0 AS A PLATFORM FOR THE ACQUISITION OF SKILLS, IMPROVE ACADEMIC PERFORMANCE AND DESIGNER CAREER PROMOTION IN THE UNIVERSITY

THE WEB 2.0 AS A PLATFORM FOR THE ACQUISITION OF SKILLS, IMPROVE ACADEMIC PERFORMANCE AND DESIGNER CAREER PROMOTION IN THE UNIVERSITY THE WEB 2.0 AS A PLATFORM FOR THE ACQUISITION OF SKILLS, IMPROVE ACADEMIC PERFORMANCE AND DESIGNER CAREER PROMOTION IN THE UNIVERSITY F. Felip Miralles, S. Martín Martín, Mª L. García Martínez, J.L. Navarro

More information

Developing an Assessment Plan to Learn About Student Learning

Developing an Assessment Plan to Learn About Student Learning Developing an Assessment Plan to Learn About Student Learning By Peggy L. Maki, Senior Scholar, Assessing for Learning American Association for Higher Education (pre-publication version of article that

More information

User education in libraries

User education in libraries International Journal of Library and Information Science Vol. 1(1) pp. 001-005 June, 2009 Available online http://www.academicjournals.org/ijlis 2009 Academic Journals Review User education in libraries

More information

1 3-5 = Subtraction - a binary operation

1 3-5 = Subtraction - a binary operation High School StuDEnts ConcEPtions of the Minus Sign Lisa L. Lamb, Jessica Pierson Bishop, and Randolph A. Philipp, Bonnie P Schappelle, Ian Whitacre, and Mindy Lewis - describe their research with students

More information

SACS Reaffirmation of Accreditation: Process and Reports

SACS Reaffirmation of Accreditation: Process and Reports Agenda Greetings and Overview SACS Reaffirmation of Accreditation: Process and Reports Quality Enhancement h t Plan (QEP) Discussion 2 Purpose Inform campus community about SACS Reaffirmation of Accreditation

More information

Concept Acquisition Without Representation William Dylan Sabo

Concept Acquisition Without Representation William Dylan Sabo Concept Acquisition Without Representation William Dylan Sabo Abstract: Contemporary debates in concept acquisition presuppose that cognizers can only acquire concepts on the basis of concepts they already

More information

A cautionary note is research still caught up in an implementer approach to the teacher?

A cautionary note is research still caught up in an implementer approach to the teacher? A cautionary note is research still caught up in an implementer approach to the teacher? Jeppe Skott Växjö University, Sweden & the University of Aarhus, Denmark Abstract: In this paper I outline two historically

More information

National and Regional performance and accountability: State of the Nation/Region Program Costa Rica.

National and Regional performance and accountability: State of the Nation/Region Program Costa Rica. National and Regional performance and accountability: State of the Nation/Region Program Costa Rica. Miguel Gutierrez Saxe. 1 The State of the Nation Report: a method to learn and think about a country.

More information

USING LEARNING THEORY IN A HYPERMEDIA-BASED PETRI NET MODELING TUTORIAL

USING LEARNING THEORY IN A HYPERMEDIA-BASED PETRI NET MODELING TUTORIAL USING LEARNING THEORY IN A HYPERMEDIA-BASED PETRI NET MODELING TUTORIAL A Paper Submitted to the Graduate Faculty of the North Dakota State University of Agriculture and Applied Science By Vaibhav Kumar

More information

School Leadership Rubrics

School Leadership Rubrics School Leadership Rubrics The School Leadership Rubrics define a range of observable leadership and instructional practices that characterize more and less effective schools. These rubrics provide a metric

More information

What Is The National Survey Of Student Engagement (NSSE)?

What Is The National Survey Of Student Engagement (NSSE)? National Survey of Student Engagement (NSSE) 2000 Results for Montclair State University What Is The National Survey Of Student Engagement (NSSE)? US News and World Reports Best College Survey is due next

More information

Rubric for Scoring English 1 Unit 1, Rhetorical Analysis

Rubric for Scoring English 1 Unit 1, Rhetorical Analysis FYE Program at Marquette University Rubric for Scoring English 1 Unit 1, Rhetorical Analysis Writing Conventions INTEGRATING SOURCE MATERIAL 3 Proficient Outcome Effectively expresses purpose in the introduction

More information

Tun your everyday simulation activity into research

Tun your everyday simulation activity into research Tun your everyday simulation activity into research Chaoyan Dong, PhD, Sengkang Health, SingHealth Md Khairulamin Sungkai, UBD Pre-conference workshop presented at the inaugual conference Pan Asia Simulation

More information

How to Judge the Quality of an Objective Classroom Test

How to Judge the Quality of an Objective Classroom Test How to Judge the Quality of an Objective Classroom Test Technical Bulletin #6 Evaluation and Examination Service The University of Iowa (319) 335-0356 HOW TO JUDGE THE QUALITY OF AN OBJECTIVE CLASSROOM

More information

Teachers Guide Chair Study

Teachers Guide Chair Study Certificate of Initial Mastery Task Booklet 2006-2007 School Year Teachers Guide Chair Study Dance Modified On-Demand Task Revised 4-19-07 Central Falls Johnston Middletown West Warwick Coventry Lincoln

More information

Kentucky s Standards for Teaching and Learning. Kentucky s Learning Goals and Academic Expectations

Kentucky s Standards for Teaching and Learning. Kentucky s Learning Goals and Academic Expectations Kentucky s Standards for Teaching and Learning Included in this section are the: Kentucky s Learning Goals and Academic Expectations Kentucky New Teacher Standards (Note: For your reference, the KDE website

More information

Case of the Department of Biomedical Engineering at the Lebanese. International University

Case of the Department of Biomedical Engineering at the Lebanese. International University Journal of Modern Education Review, ISSN 2155-7993, USA July 2014, Volume 4, No. 7, pp. 555 563 Doi: 10.15341/jmer(2155-7993)/07.04.2014/008 Academic Star Publishing Company, 2014 http://www.academicstar.us

More information

Rendezvous with Comet Halley Next Generation of Science Standards

Rendezvous with Comet Halley Next Generation of Science Standards Next Generation of Science Standards 5th Grade 6 th Grade 7 th Grade 8 th Grade 5-PS1-3 Make observations and measurements to identify materials based on their properties. MS-PS1-4 Develop a model that

More information

TEACHING SECOND LANGUAGE COMPOSITION LING 5331 (3 credits) Course Syllabus

TEACHING SECOND LANGUAGE COMPOSITION LING 5331 (3 credits) Course Syllabus TEACHING SECOND LANGUAGE COMPOSITION LING 5331 (3 credits) Course Syllabus Fall 2009 CRN 16084 Class Time: Monday 6:00-8:50 p.m. (LART 103) Instructor: Dr. Alfredo Urzúa B. Office: LART 114 Phone: (915)

More information

IST 649: Human Interaction with Computers

IST 649: Human Interaction with Computers Syllabus for IST 649 Spring 2014 Zhang p 1 IST 649: Human Interaction with Computers Spring 2014 PROFESSOR: Ping Zhang Office: Hinds Hall 328 Office Hours: T 11:00-12:00 pm or by appointment Phone: 443-5617

More information

Book Review: Build Lean: Transforming construction using Lean Thinking by Adrian Terry & Stuart Smith

Book Review: Build Lean: Transforming construction using Lean Thinking by Adrian Terry & Stuart Smith Howell, Greg (2011) Book Review: Build Lean: Transforming construction using Lean Thinking by Adrian Terry & Stuart Smith. Lean Construction Journal 2011 pp 3-8 Book Review: Build Lean: Transforming construction

More information

SY 6200 Behavioral Assessment, Analysis, and Intervention Spring 2016, 3 Credits

SY 6200 Behavioral Assessment, Analysis, and Intervention Spring 2016, 3 Credits SY 6200 Behavioral Assessment, Analysis, and Intervention Spring 2016, 3 Credits Instructor: Christina Flanders, Psy.D., NCSP Office: Samuel Read Hall, Rm 303 Email: caflanders1@plymouth.edu Office Hours:

More information

Notes on The Sciences of the Artificial Adapted from a shorter document written for course (Deciding What to Design) 1

Notes on The Sciences of the Artificial Adapted from a shorter document written for course (Deciding What to Design) 1 Notes on The Sciences of the Artificial Adapted from a shorter document written for course 17-652 (Deciding What to Design) 1 Ali Almossawi December 29, 2005 1 Introduction The Sciences of the Artificial

More information

Quality teaching and learning in the educational context: Teacher pedagogy to support learners of a modern digital society

Quality teaching and learning in the educational context: Teacher pedagogy to support learners of a modern digital society Journal of Student Engagement: Education Matters Volume 2 Issue 1 Article 13 2012 Quality teaching and learning in the educational context: Teacher pedagogy to support learners of a modern digital society

More information

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS 1 CALIFORNIA CONTENT STANDARDS: Chapter 1 ALGEBRA AND WHOLE NUMBERS Algebra and Functions 1.4 Students use algebraic

More information

Software Maintenance

Software Maintenance 1 What is Software Maintenance? Software Maintenance is a very broad activity that includes error corrections, enhancements of capabilities, deletion of obsolete capabilities, and optimization. 2 Categories

More information

A Case-Based Approach To Imitation Learning in Robotic Agents

A Case-Based Approach To Imitation Learning in Robotic Agents A Case-Based Approach To Imitation Learning in Robotic Agents Tesca Fitzgerald, Ashok Goel School of Interactive Computing Georgia Institute of Technology, Atlanta, GA 30332, USA {tesca.fitzgerald,goel}@cc.gatech.edu

More information