Combining and selecting characteristics of information use

Size: px
Start display at page:

Download "Combining and selecting characteristics of information use"

Transcription

1 Combining an selecting characteristics of information use Ian Ruthven *1, Mounia Lalmas 2 an Keith van Rijsbergen 3 1 Department of Computer an Information Sciences, University of Strathclye, Glasgow, G1 IXH 2 Department of Computer Science, Queen Mary, University of Lonon, Lonon, E1 4NS 3 Department of Computing Science, University of Glasgow, Glasgow, G12 8QQ Ian.Ruthven@cis.strath.ac.uk, mounia@cs.qmul.ac.uk, keith@cs.gla.ac.uk Abstract In this paper we report on a series of experiments esigne to investigate the combination of term an ocument weighting functions in Information Retrieval. We escribe a series of weighting functions, each of which is base on how information is use within ocuments an collections, an use these weighting functions in two types of experiments: one base on combination of evience for a-hoc retrieval, the other base on selective combination of evience within a relevance feeback situation. We iscuss the ifficulties involve in preicting goo combinations of evience for a-hoc retrieval, an suggest the factors that may lea to the success or failure of combination. We also emonstrate how, in a relevance feeback situation, the relevance assessments can provie a goo inication of how evience shoul be selecte for query term weighting. The use of relevance information to guie the combination process is shown to reuce the variability inherent in combination of evience. 1 Introuction Most relevance feeback (RF) algorithms attempt to bring a query closer to the user s information nee by reweighting or moifying the terms in a query. The implicit assumption behin these algorithms is that we can fin an optimal combination of weighte terms to escribe the user s information nee at the current stage in a search. However, relevance as a user jugement is not necessarily ictate only by the presence or absence of terms in a ocument. Rather relevance is a factor of what concepts the terms represent, the relations between these concepts, how users interpret them an how they relate to the information in the ocument. From stuies such as those conucte by Barry an Schamber, [BS98], it is clear that current moels of RF, although successful at improving recall-precision, are not very sophisticate in expressing what makes a ocument relevant to a user. Denos et al., [DBM97] for example, make the goo point that although users can make explicit jugements on why ocuments are relevant, most systems cannot use this information to improve a search. Not only are users' jugements affecte by a variety of factors but they are base on the ocument text. RF algorithms, on the other han, typically are base on a representation of a text an only consier frequency information or the presence or absence of terms in ocuments. These algorithms o not look eeper to see what it is about terms that inicate relevance; they ignore information on how the term is use within ocuments. For example a ocument may only be relevant if the terms appear in a certain context, if certain combinations of terms occur or if the main topic of the ocument is important. Extening feeback algorithms to incorporate the usage of a term within ocuments woul not only allow more precise querying by the user but also allows relevance feeback algorithms to aapt more subtly to users relevance jugements. In this paper we investigate how incorporating more information on the usage of terms can improve retrieval effectiveness. We examine a series of term an ocument weighting functions in combination an in selective combination: selecting which characteristics of a term (e.g. frequency, context, istribution within ocuments) or ocument (complexity, ratio of useful information) shoul be use to retrieve ocuments. This research extens the initial stuy presente in [RL99] which emonstrate that a subset of the weighting function use in this paper were successful for precision enhancement. In particular we investigate the role of combination of evience in RF. * Corresponing author. This work was complete while the first author was at the University of Glasgow. 1

2 The following sections outline how we escribe term an ocument characteristics (section 2), the ata we use in our experiments (section 3), our experimental methoology (section 4), an the results of three sets of experiments. The first set of experiments examines each characteristic as a single retrieval function (section 5). The secon set looks at combining evience of term use in stanar retrieval (section 6), an the thir set examines selecting evience in RF (sections 7 an 8). We summarise our finings in section 9. 2 Term an ocument characteristics In this section we outline five alternative ways of escribing term importance in a ocument or collection - or term characteristics. Three of these are stanar term weighting functions, if, tf an noise, another two are evelope for this work. inverse ocument frequency, base on how often a term appears within a collection, escribe in section 2.1 noise, also base on how often a term appears within a collection but base on within-ocument frequency, section 2.2 term frequency, base on how often a term appears within a ocument, section 2.3 thematic nature, or theme, base on how a term is istribute within a ocument, section 2.4 context, base on the proximity of one query term to another query term within the same ocument, section 2.5 In aition, we introuce two ocument characteristics. These escribe some aspect of a ocument that ifferentiates it from other ocuments. specificity, base on how many unique terms appear in a ocument, section 2.6. information-noise, base on the proportion of useful to non-useful content within a ocument, section 2.7 These characteristics were chosen to be representative of general weighting schemes those that represent information on general term appearance, e.g. if, tf, - an specific weighting schemes those that represent specific features on how terms appear in ocuments, e.g. theme. 2.1 if Inverse ocument frequency, or if, [SJ72] is a stanar IR term weighting function that measures the infrequency, or rarity, of a term's occurrence within a ocument collection. The less likely a term is to appear in a collection the better is it likely to be at iscriminating relevant from irrelevant ocuments. In these experiments we measure if by the equation shown in equation 1. n if ( t) = log + 1 N Equation 1: inverse ocument frequency (if) where n is the number of ocuments containing the inexing term t an N is the number of ocuments in the collection 2.2 noise The secon term characteristic we investigate was the noise characteristic iscusse in [Sal83, Har86], equation 2. The noise characteristic gives a measure of how important a term is within a collection but unlike if, noise is base on within-ocument frequency. noise ( t) = k N i= 1 frequency ik total _ frequency k total _ frequency log frequency ik k Equation 2: noise where N = number of ocuments in the collection, frequency ik = the frequency of term k in ocument i, total_frequency k = total frequency of term k in the collection From equation 2, if a term appears in only one ocument, it receives a noise score of zero. Terms that appear more commonly throughout a collection receive a higher noise value. The noise value is then inversely proportional to its iscrimination power. The noise characteristic as efine here therefore requires 2

3 normalisation, [Har86], to ensure that the noise value of a term reflects its iscriminatory power. To normalise the noise score, we subtracte the noise score of a term from the maximum noise score. The normalise noise characteristic gives a maximum noise score to a term if all its occurrences appear in one ocument an the lowest noise score if all occurrences of the term appear in ifferent ocuments. 2.3 tf Incluing information about how often a term occurs in a ocument - term frequency (tf) information - has often been shown to increase retrieval performance, e.g. [Har92]. For these experiment we use the following formula, tf ( t) = log( occurrences ( ) 1) / log( occurrences ( )) t + Equation 3: term frequency (tf) where occurrences t () occurrences t () is the number of occurrences of term t in ocument, occurrences total () is the total number of term occurrences in ocument. 2.4 theme Previous work by for example Hearst an Plaunt [HP93] an Parais an Berrut [PB96], emonstrate that taking into account the topical or thematic nature of ocuments can improve retrieval effectiveness. Hearst an Plaunt present a metho specifically for long ocuments, whereas Parais an Berrut's metho is base on precise conceptual inexing of ocuments. We present a simple term-base alternative base on the istribution of term occurrences within a ocument. This is base on the assumption that the less evenly istribute the occurrences of a term are within a ocument, then the more likely the term is to correspon to a localise iscussion in the ocument, e.g. a topic in one section of the ocument only, Figure 1 left-han sie. Conversely, if the term s occurrences are more evenly sprea throughout the ocument, then we may assume that the term is somehow relate to the main topic of the ocument, Figure 1 right-han sie. Unlike Hearst an Plaunt we o not split the ocument into topics an assign a sub- or main-topic classification. Instea we efine a theme value of a term, which is base on the likelihoo of a term to be a main topic. total Figure 1: Localise iscussion of term X (left-han sie), general iscussion of term X (right-han sie) The algorithm which we evelope for this is shown in equation 4. This value is base on the ifference between the position of each occurrence of a term an the expecte positions. Table 1 gives a short example for a ocument with 1000 wors, an five occurrences of term t. First, we calculate whether the first occurrence of term t occurs further into the ocument that we woul expect, base on the expecte istribution (first(t) - line two, equation 1; Column 7, Table 1). Next we calculate whether the last occurrence of the term appears further from the en of the ocument than we woul expect (last(t) - line two, equation 1; Column 8, Table 1). For the remainer of the terms we calculate the ifference between the expecte position of a term, base on the actual 3

4 position of the last occurrence an the expecte ifference between two occurrences ( line two; Column 4-6, n 1 Table 1) ( i= 2 preicte _ position ( t) actual _ position ( t) ). i theme ( t) = ( length ifference ( t)) / length i where ifference ( t) = first ( t) + last ( t) + n 1 i= 2 preicte _ position ( t) actual _ position ( t) i i first ( t) = 0, if actual _ position1( t) istribution ( t) = actual _ position1 ( t) istribution ( t), ow last ( t) = 0, if ( length actual _ position ( t) istribution ( t)) = length ( actual _ positionn + istribution ( t)), ow preicte _ positioni = actual _ positioni 1 + istribution ( t) istribution ( t) length / occurrences ( t) = Equation 4: theme characteristic where istribution(t) is the expecte istribution of term t in ocument, assuming all occurrences of t are equally istribute, preicte_positioni is the expecte position of the ith occurrence of term t, actual_positioni is the actual position of the ith occurrence. occurrences (t) is the number of occurrences of term t in ocument. n is the number of query terms in the ocument. n length occs istr epos pos iff first last ifference theme Table 1: Example calculation of theme value for a term We then sum these values to get a measure of the ifference between the expecte position of the term occurrences an their actual positions. The greater the ifference between where term occurrences appear an where we woul expect them to appear, the smaller the theme value for the term. The smaller the ifference, the larger the theme value for the term. 2.5 context There are various ways in which one might incorporate information about the context of a query term. For example, we might rely on coocurrence information [VRHP81], information about phrases [Lew92], or information about the logical structures, e.g. sentences, in which the term appears [TS98]. We efine the importance of context to a query term as being measure by its istance from the nearest query term, relative to the average expecte istribution of all query terms in the ocument. 4

5 context( t) = ( istribution ( q) min ( t)) / istribution ( q) min ( t) = min ' ( position ( t) position ( t' )) t t istribution ( q) length / occurrences ( q) = Equation 5: context characteristic for term t in ocument where istribution (q) is the expecte istribution of all query terms in the ocument, assuming terms are istribute equally position (t) is the position of term t an min (t) is the minimum ifference from any occurrence of term t to another, ifferent query term. 2.6 specificity The first ocument characteristic we propose is the specificity characteristic which is relate to if. The if characteristic measures the infrequency of a term's occurrence within a ocument collection; the less likely a term is to appear in a ocument the better is it likely to be at iscriminating relevant from irrelevant ocuments. However, if oes not consier the relative iscriminatory power of other terms in the ocument. If a ocument contains a higher proportion of terms with a high if, it may be more ifficult to rea, e.g. if it contains a lot of technical terms. On the other han a ocument containing a lot of terms with very low if values may contain too few information-bearing wors. We propose the specificity characteristic as a measure of how specialise a ocument s contents are, relative to the other ocuments in the collection. This is a very simple measure as we o not take into account the omain of the ocument or external knowlege sources, which woul allow us to represent the complexity of the ocument base on its semantic content. The specificity characteristic is a ocument characteristic, giving a score to an entire ocument rather than iniviual terms. It is measure by the sum of the if values of each term in the ocument, ivie by the number of terms in the ocument, giving an average if value for the ocument, equation 6. specificity( ) = n i n if ( i) Equation 6: specificity ocument characteristic of ocument where n = number of terms in ocument 2.7 information-to-noise The specificity characteristic measure the complexity of the ocument base on if values. An alternative measure is the information-to-noise ratio, [ZG00], abbreviate to info-noise. This is calculate as the number of tokens after processing (stemming an stopping) of the ocument ivie by the length of the ocument before stopping an stemming, equation 7. info_noise() = processe _ length( ) length( ) Equation 7: info_noise ocument characteristic of ocument where processe_length() = number of terms in ocument after stopping an stemming length() = number of terms in ocument before stopping an stemming info_noise, as escribe in [ZG00], measures the proportion of useful to non-useful information content within a ocument. 2.8 Summary The if an noise characteristics give values to a term epening on its importance within a collection, the tf an theme characteristics give values epening on the term s importance within an iniviual ocument an the specificity an info_noise characteristics give values to iniviual ocuments base on their content. The context characteristic gives a value to a term base on its proximity to another query term in the same ocument. Each of 5

6 the term characteristics can be use to ifferentiate ocuments base on how terms are use within the ocuments an the ocument characteristics allow ifferentiation of ocuments base on their content. The ocument characteristics also allow RF algorithms to base feeback ecisions on the ocument taken as a whole, rather than only iniviual components of the ocument. Each of the algorithms that calculate the characteristic values give scores in ifferent ranges. In our experiments we scale all values of the characteristics to fall within the same range, 0-50, to ensure that we were working with comparable values for each characteristic. In the next section we outline the ata we use in our experiments. 3 Data For the experiments reporte in this paper we use two sets of collections. The first is a set of three small test collections (CACM, CISI an MEDLARS collections 1 ), the secon is a set of two larger collections (the Associate Press (1988) (AP) an the Wall Street Journal ( ) (WSJ) collection from the TREC initiative [VH96]). Statistics of these collections are given in Table 2. CACM CISI MEDLARS AP WSJ Number of ocuments Number of queries use Average ocument length Average wors per query Average relevant ocuments per query Number of unique terms in the collection Table 2: Details of CACM, CISI, MEDLARS, AP an WSJ collections The AP an WSJ test collections each come with fifty so-calle TREC topics. Each topic escribes an information nee an those criteria that were use in assessing relevance when the test collection was create. A TREC topic has a number of sections, (see Figure 2 for an example topic). In our experiments we only use the short Title section from topics as queries, as using any more of the topic escription may be an unrealistic as a user query. Number: 301 Title: International Organize Crime Description: Ientify organisations that participate in international criminal activity, the activity, an, if possible, collaborating organisations an the countries involve. Narrative: A relevant ocument must as a minimum ientify the organisation an the type of illegal activity (e.g., Columbian cartel exporting cocaine). Vague references to international rug trae without ientification of the organisation(s) involve woul not be relevant. Figure 2: Example of a TREC topic Stopwors were remove, using the stopwor list in [VR79], an the collections were stemme using the Porter stemming algorithm [Por80]. 4 Outline of experiments In this paper we escribe three sets of experiments: Each collection comes with a number of queries. However, for some queries there are no relevant ocuments in the collection. As these queries cannot be use to calculate recall-precision figures they are not use in these experiments. This row shows the number of queries, for each collection, for which there is at least one relevant ocument. 3 After the application of stemming an stopwor removal. 4 This row shows the average length of the queries that were use in the experiments. 6

7 i. retrieval by single characteristic. In section 5 we present results obtaine by running each characteristic as a single retrieval function. In this section we examine the relative performance of each characteristics on the test collections, an iscuss why some characteristics perform better than others as retrieval functions. ii. retrieval by combination of characteristics. In section 6 we investigate whether combining characteristics can improve retrieval effectiveness over retrieval by single characteristic. We also iscuss factors that affect the success of combination, such as the size of the combination an which characteristics are combine. iii. relevance feeback. In section 7 we investigate how we can use relevance assessments to select goo combinations of characteristics of terms an ocuments to use for relevance feeback. We escribe several methos of selecting which characteristics are important for a query an compare these methos against methos that o not use selection of characteristics. 5 Retrieval by single characteristic In this section we examine the performance of running each characteristic (term an ocument characteristics) as a single retrieval function (retrieval by the sum of the if value of each query term, retrieval by the sum of tf values of each query term, etc.). The results are presente in section 5.2 but before this, in section 5.1, we look at how ocument characteristics shoul be use to score ocuments. 5.1 Document characteristics - initial investigations As the specificity an info-noise characteristics are ocument rather than term characteristics, they give a value to each ocument irrespective of which terms are in the query. However, we can use the ocument characteristics to prouce ifferent rankings base on two criteria: i. which ocuments receive a score. Although all ocuments have a value for the specificity an infonoise characteristics, we may choose to score only those ocuments that contain at least one query term, as these ocuments are those that are the most likely to be relevant. To investigate this, we assesse two methos of retrieving ocuments - the query epenent - an the query inepenent strategies. In the query inepenent strategy the retrieval score of a ocument is the characteristic score (info_noise or specificity). This metho gives an ientical ranking of ocuments for all queries. In the query epenent strategy the retrieval score of a ocument is also the characteristic score but this score is only assigne to those ocuments that contain at least one query term. If the ocument contains no query terms then the retrieval score is zero. In this metho we, then, retrieve all ocuments that contain a query term before the ocuments that contain no query terms, giving a ifferent ranking to each query. ii. how to orer the ocuments. The specificity characteristic gives high scores to more complex ocuments, whereas the info_noise characteristic gives high scores to ocuments that have a high proportion of useful information. This means that we are asserting that relevant ocuments are more likely to have a higher amount of useful information or a higher complexity. This requires testing. We teste two strategies - stanar - in which we rank ocuments in ecreasing orer of characteristic score an reverse - in which we rank ocuments in increasing orer of characteristic score. These two criteria give us four combinations of strategy - query epenent an stanar, query inepenent an stanar, query epenent an reverse, query epenent an reverse. Each of these strategies correspon to a ifferent metho of ranking ocuments. The results of these ranking strategies are shown in Table 3 for the specificity characteristic an Table 4 for the info_noise characteristic 5. Also shown in each table, for comparison, are the results of two ranom retrieval runs on each collection 6. These are also base on a query epenent strategy (ranom orer of all ocuments containing a query term, followe by ranom orer of the remaining ocuments) an a query inepenent strategy (a completely ranom orering of all ocuments). 5 Full recall-precision tables for all experiments are given in an electronic appenix, available at The corresponing tables will be note as footnotes throughout the paper. Appenix A, Tables A.3 A Tables A.1 an A.2. 7

8 stanar specificity reverse specificity ranom Collection query epenent query inepenent query epenent query inepenent query epenent query inepenent CACM CISI MEDLARS AP WSJ Table 3: Average precision figures for specificity characteristic Highest average precision figures for each collection are shown in bol From Table 3, the specificity characteristic is best applie using a query epenent strategy. Whether or not it is applie in ecreasing orer of characteristic value (stanar), or increasing orer of characteristic score (reverse) is collection epenent. However the overall preference is for the reverse strategy. From Table 4 the info_noise characteristic is best applie using the query-epenent stanar strategy: orering ocuments containing a query term an with the highest proportion of useful information at the top of the ranking. stanar info_noise reverse info_noise ranom Collection query epenent query inepenent query epenent query inepenent query epenent query inepenent CACM CISI MEDLARS AP WSJ Table 4: Average precision figures for info_noise characteristic Highest average precision figures for each collection are shown in bol On all collections, except the MEDLARS collection, at least one metho of applying the characteristics gave better performance than ranom (query inepenent), an with the exception of MEDLARS an CISI also performe better than the query epenent ranom run. One possible reason for the poorer results on these collections is that the range of ocument characteristic values for these collections is not very wie. Consequently the characteristics o not have enough information to iscriminate between ocuments. It is better to rank only those ocuments that contain a query term than all ocuments. This is not surprising as, using the query epenent strategy, we are in fact re-ranking the basic if ranking for each query. We shall iscuss the relative performance of the ocument characteristics against the term characteristics in the next section. Although the ocument characteristics o not give better results than the term characteristics (see next section), they o generally give better results than the ranom retrieval runs an can be use in combination to ai retrieval. 5.2 Single retrieval on all characteristics The results from running each characteristic as a single retrieval function are summarise in Table 5 7, measure against the query epenent ranom strategy. This is use as a baseline for this experiment as all the characteristics prioritise retrieval of ocuments that contain a query term over those ocuments that contain no query terms. Hence this metho of running a ranom retrieval is more similar in nature to the term characteristics an, as it gives higher average precision, provies a stricter baseline measure for comparison. The majority of characteristics outperform the query epenent ranom retrieval baseline. However some characteristics o perform more poorly than a ranom retrieval of the ocuments (info_noise on CISI, theme, specificity an info_noise on MEDLARS, context on WSJ) 8. 7 Tables A.14 A All characteristics, for all collections except MEDLARS, outperforme a completely ranom retrieval. 8

9 Characteristic Collection if tf theme context specificity noise info-noise ranom CACM CISI MEDLARS AP WSJ Table 5: Average precision figures for term an ocument characteristics use as single retrieval functions Highest average precision figures for each collection are shown in bol The orer in which the characteristics performe is shown in Figure 3 where > inicates statistical significance an >= inicates non-statistical significance. 9 The ocument characteristics perform quite poorly as they are insensitive to query terms. That is, although, when using these characteristics we score only ocuments that contain a query term, the ocument characteristics o not istinguish between ocuments that contain goo query terms an ocuments that contain poor query terms. CACM noise >= tf >= if > context >= theme > inf_n > spec > ranom CISI tf > if > noise > spec > context > theme > ranom >= inf_n MEDLARS noise >= tf >= if > context > ranom > theme > inf_n > spec AP if > tf >= context > theme > noise > spec >= inf_n > ranom WSJ if > tf > noise >= theme >= spec >= inf_n > ranom > context Figure 3: Statistical an non-statistical ifferences between characteristics on all collections where spec = specificity, inf_n = info_noise On nearly all collections the stanar characteristics (if, tf, noise 10 ) outperforme our new characteristics. One possible reason for this is that, although, the new term characteristics (theme, context) give a weight to every term in a ocument, unlike the stanar characteristics they o not always give a non-zero weight. The context characteristic, for example, will only assign a weight to a term if at least two query terms appear in the same ocument. In the case of the two larger collections we have relatively smaller queries. Hence the co-occurrence of query terms within a ocument may be low with the resulting effect that most terms have a zero weight for this characteristic. This, in turn, will lea to a poor retrieval result as the characteristic cannot istinguish well between relevant an non-relevant ocuments. Similarly, the theme characteristic, as implemente here, will also lea a high proportion of terms being assigne a zero weight compare with the tf characteristic. One reason for this is that theme assigns a zero weight to a term if it only appears once within a ocument. A collection such as the MEDLARS collection, which has a high number of terms that only appear in one ocument may be more susceptible to this, as it contains a large number of unique terms. The stanar characteristics are also less strict algorithms: the information they represent, e.g. frequency of a term within a ocument, is more general than that represente by the new characteristics. This will mean that the stanar characteristics will be useful for a wier range of queries. For example, tf will be a useful characteristic for most query terms as, generally, the more often a query term appears within a ocument, the more likely the ocument is to be relevant. The theme characteristic, on the other han, will only be useful for those queries where the query terms are relate to the main topic of the ocument. For queries where this conition is not met, the theme characteristic will not be useful. Even though the new characteristics o not perform as well as the traitional weighting functions they o improve retrieval effectiveness over ranom retrieval. These algorithms shoul not be seen as alternative weighting schemes but as aitional ones: ones that provie aitional methos of iscriminating relevant from non-relevant material. In RF these aitional characteristics will be use to score query terms if they are useful at inicating relevant ocuments for iniviual queries. That is, by proviing evience of ifferent aspects of 9 Calculate using a paire t-test, p < 0.05, holing recall fixe an varying precision 10 Harman s, [Har86], experimental investigation of the noise term weighting function on the Cranfiel collection showe superior results for noise over if. In these experiments, this hel for the shorter CACM an MEDLARS collection. However in the larger collections, the noise characteristic performe relatively poorly. 9

10 information use, they can be use to help retrieval performance in combination with other characteristics. This combination of evience is the subject of the next section. 6 Retrieval by combination of characteristics In the previous section we looke at the performance of each characteristic iniviually. In this section we look at whether the retrieval effectiveness of characteristics will be improve if we use them in combination. Belkin et al., [BKFS95], examine the role of multiple query representations in a-hoc IR. Their argument in favour of ifferent representations of queries is twofol: i. empirical evience that ifferent retrieval functions retrieve ifferent ocuments, e.g. [Lee98]. In our approach combinations of ifferent representations of query terms can retrieve ocuments that fulfil ifferent criteria for relevance. ii. theoretical: ifferent query representations can provie ifferent interpretations of a user's unerlying information nee. This has a strong connection to Ingwersen's work on polyrepresentation - multiple representations of the same object, in our case a query, rawn from ifferent perspectives can provie better insight into what constitutes relevance than a single goo representation, [Ing94]. In this experiment we teste all possible combinations of the characteristics, running each possible combination as a retrieval algorithm. For each collection, we effectively run the powerset of combinations, each set comprising a ifferent combination of characteristics. For each combination, the retrieval score of a ocument was given by sum of the score of each characteristic of each query term that occurre in the ocument. For example, for the combination of tf an theme, the score of a ocument was equal to the sum of the tf value of each query term plus the sum of the theme value of each query term. Two versions of this experiment were run, the first use the values of characteristics given at inexing time, the secon treate the characteristics as being more or less important than each other. There are several reasons for this. For example, some characteristics may reflect aspects of information use that are more easily measure than another; others are better as retrieval functions an shoul be treate as being more important. We incorporate this by introucing a set of scaling weights (if 1, tf 0.75, theme 0.15, context 0.5, noise 0.1, specificity an information_noise ) that are use to alter the weight given to a term at inexing time. Each inexing weight of a term characteristic is multiplie by the corresponing scaling weight, e.g. all tf values are multiplie by 0.75, all theme values by 0.15, etc. This gives us two conitions - weighting an non-weighting of characteristics - for each combination of characteristics. In the following sections we shall summarise our finings regaring three aspects: the effect on retrieval effectiveness of combining characteristics, the effect of weighting characteristics, an the effect of aing iniviual characteristics to other combinations. Each of these will be iscusse in a separate section in sections We shall summarise in section Effecting of combining characteristics Our experimental hypothesis is that combining characteristics can increase retrieval effectiveness over using iniviual characteristics. In section 6.3 we shall iscuss how well the iniviual characteristics performe in combination. In this section we shall examine the basic hypothesis an iscuss general finings. In Table 6 we outline the effect on iniviual characteristic performance by the aition of other characteristics. Of the 127 possible combinations of characteristics for each collection, each characteristic appeare in 63 combinations. Each row is a count of how many of these 63 combinations containing each characteristic ha higher average precision (increase) than the characteristic as a single retrieval function, lower average precision (ecrease), or no change in average precision (none). For example, how many combinations containing if gave an average precision figure that was better, worse or ientical to the average precision of if alone? The first general conclusion from Table 6 is that all characteristics can benefit from combination with another characteristic or set of characteristics. Furthermore, with the exception of the noise characteristic on the CACM, 11 These weights were erive from experiments using a sample of the ata from each collection. 12 Tables A.19 A

11 an the tf an if characteristics on the CISI, any characteristic was more likely to benefit from combination than be harme by it. This conclusion hel uner both the weighing an non-weighting conitions. The secon general conclusion is that the performance of a characteristic as a single retrieval function (section 4.2) is a goo inicator of how well the characteristic will perform in combination. The poorer the characteristic is at retrieving relevant ocuments the more likely it is to benefit from combination with another characteristic. For each collection, on the whole, the poorer characteristics 13 improve more often in combination with other characteristics. The reverse also hols: if a characteristic is goo as a single retrieval function, then there is less chance that it will be improve in combination. For example the best characteristics in the small collections (tf, if on CISI, an noise on CACM) showe the lowest overall improvement in combination. However the overall tenency is beneficial: combination benefits more characteristics than it harms. Collection Conition Change if tf theme context spec noise info_ noise CACM NW increase ecrease none W increase ecrease none CISI NW increase ecrease none W increase ecrease none MEDLARS NW increase ecrease none W increase ecrease none AP NW increase ecrease none W increase ecrease none WSJ NW increase ecrease none W increase ecrease none Table 6: Effect of combination on iniviual characteristics where increase = increase in average precision when combine, ecrease = ecrease in average precision when in combination, none = no ifference in average precision when in combination, NW = non-weighting conition, W = weighting conition Bol figures inicate the preominant effect of the characteristic in combination In the remainer of this section we look at what affects the success of combination. In particular, we look examine the size of combinations an the components of combinations. 13 These were the theme, context, specificity an info_noise for the CACM, CISI an MEDLARS collections an theme, context, noise, specificity an info_noise for the AP an WSJ collections. 11

12 In Table 7 we analyse the success of combination by size of combination, that is how many characteristics were combine. For each conition, weighting an non-weighting, on each collection we ranke all combinations by average precision 14. We then took the meian 15 value an the size of the combinations that appeare above an below this point. In Table 7 bol figures inicate where most combinations, of a given size, appeare (above or below the meian point). In the majority of cases the larger combinations (combinations of 4-7 characteristics) performe better than the meian value, an the smaller combinations (combinations of 1-3 characteristics) performe worse than the meian. There was little ifference between the weighting an non-weighting conitions. One possible reason for the success of the larger combinations is that poor characteristics have a lower overall effect in a larger combination. That is, if we only combine two characteristics an one of these is a poor characteristic, then there is a greater chance that the combination will perform less well than the better iniviual characteristic. Conversely, if we combine a number of characteristics, an one is poorer than the rest, then this will not have such a great effect on the performance of the combination. Collection Position Conition CACM Above NW W Below NW W CISI Above NW W Below NW W MEDLARS Above NW W Below NW W AP Above NW W Below NW W WSJ Above NW W Below NW W Table 7: Distribution of combinations over ranking of average precision where Above = combination falls above or at meian point of ranking, Below = combination falls below meian point of ranking, NW = non-weighting conition, W = weighting conition A further reason for larger combinations performing more effectively is that they allow for a more istinct ranking. That is, the more methos we have of scoring ocuments, the less chance that ocuments will receive an equal retrieval score. Now we look at how the components of the combinations affect the success of combining characteristics. As state before, each characteristic appeare in a total of 63 combinations. Table 8 presents how many of these combinations appeare above the meian combination in the ranking of average precision, i.e. how many times a combination containing a characteristic performe better than average. The better iniviual characteristics, e.g. if an tf, appeare in more combinations above the meian than below for all collections. The poorer characteristics, e.g. info_noise, tene to appear in more combinations below the meian than above. 14 Tables A.131 A.141, in 15 For each collection, in each conition, there were 127 possible combinations, the meian point was taken to be the 64 th combination in the ranking of all combinations. 12

13 CACM CACM CISI CISI MEDLARS MEDLARS AP AP WSJ WSJ NW W NW W NW W NW W NW W if tf theme context spec noise inf Table 8: Number of appearances of a characteristic in a combination appearing above meian combination Bol figures inicate where the majority of the combinations containing an iniviual characteristic appeare above the meian value. This is not necessarily to say, however, that poor characteristics always ecrease the performance of a combination, see section 6.4 for example. Often a characteristic that performs less well as a single characteristic can improve a combination. What is important is how well a combination of characteristics separates relevant from irrelevant ocuments for an iniviual query: a particular combination may work poorly on average but work well for certain queries. This is important for our RF experiments, in which we select which are goo characteristics for iniviual queries, section 7. To summarise our finings: combinations of characteristics, whether weighte or not, is beneficial for all characteristics on all collections teste. This benefit is greater when the characteristic is poor as a single retrieval function but the overall benefits of combination still hols for goo characteristics. The larger combinations (4-7 characteristics) ten to be better than small (1-3 characteristics) as retrieval functions over the collections. 6.2 Effect of weighting characteristics Our basis behin weighting characteristics was that some characteristics may be better at inicating relevance than others. In Table 9 we summarise the effect of weighting on each collection, inicating the number of combinations that increase/ecrease in average precision when using weighting. Overall, 47% of combinations improve using weighting on CACM collection, 61% on CISI, 60% MEDLARS, 69% on AP an 66% on WSJ. As can be seen for all collections, except CACM, weighting was beneficial in that it improve the average precision of more combinations than it ecrease. Generally these improvements were statistically significant. Increase Decrease Collection Significant Nonsignificansignificant Significant Non- CACM 24 20% 32 27% 31 26% 33 28% CISI 59 49% 14 12% 37 31% 10 8% MEDLARS 45 38% 27 23% 23 19% 25 21% AP 51 43% 32 27% 22 18% 15 13% WSJ 67 56% 12 10% 26 22% 15 13% Table 9: Effect of weighting on combination performance Significant = statistically significant change, Non-significant = non statistically significant change Bol figures inicate preominant effect of weighting on each collection Table 10 breaks own these figures by size of combination, the number of characteristics in the combination. The combination that benefite most from weighting were also these tene to be the ones that performe best in combination, i.e. those combination of four or greater characteristics. 13

14 Collection Change CACM Increase Decrease CISI Increase Decrease MEDLARS Increase Decrease AP Increase Decrease WSJ Increase Decrease Table 10: Effect of weighting by size of combination Bol figures inicate preominant effect on each size of combination In Table 11, we analyse which characteristics appeare in the combinations that i better using weighting than no weighting. Generally, combinations containing if an tf were helpe by weighting across the collections an theme an context were helpe in the larger collection. The only characteristic to be consistently harme by weighting was the noise characteristic. if tf theme context spec noise info_noise CACM % 75% 61% 41% 59% 32% 46% CISI % 67% 37% 44% 58% 29% 52% MEDLARS % 56% 40% 49% 64% 13% 67% AP % 55% 66% 54% 48% 18% 58% WSJ % 57% 62% 57% 49% 25% 49% Table 11: Appearance of iniviual characteristics in combinations that were improve by weighting Bol figures inicate those characteristics for which weighting was beneficial overall. Weighting is generally beneficial but it is important to get goo values for the characteristics. For example, both if an tf were goo iniviual retrieval algorithms an were highly weighte which helpe their performance in combination as the combination was more heavily biase towars the ranking given by these characteristics. noise, on the other han, was a variable retrieval algorithm in that it performe well on some collections an more poorly on others. As it was weighte lowly the overall effect of noise in combination was lessene in the weighting conition. Consequently in cases where noise woul have been a goo iniviual retrieval algorithm the combination i not perform as well as it might have without weighting. A final observation is that although weighting i not generally improve the best combination for the collections 16, it i ten to improve the performance of the mile ranking combinations significantly. These were the combinations that appeare in the mile of the ranking of combinations escribe in section 6.1. Weighting then was a success in that it improve the performance of most combinations. However it achieve this by ecreasing the performance of the poorer combinations an increasing the performance of the average combinations. 16 Tables A.131 A

15 6.3 Effect of aing iniviual characteristics In section 6.1 we gave general conclusions about the effect of combining characteristics. In this section we look more closely at the effect of combining iniviual characteristics an the effect of characteristics on the performance of a combination of characteristics. In Table 12 we summarise the effect of aing a characteristic to other combinations, e.g. aing if to the 63 combinations that i not alreay contain if. We measure whether the new information causes an increase in average precision (aing if improves retrieval), a ecrease in average precision (aing if worsens retrieval), or no change in average precision (aing if gives the same retrieval effectiveness). We look first at the aition of iniviual characteristics to any combination of other characteristics. On all collections the aition of if or tf information to a combination of characteristics was beneficial. This was more pronounce in the larger AP an WSJ collections, an hel uner both the weighting an nonweighting conitions. The aition of theme information improves the performance of other combinations in smaller collections using either weighting or non-weighting. In the larger collections, the theme characteristic only improve performance uner the weighting conition. CACM CISI MEDLARS AP WSJ No Wgt No Wgt No Wgt No Wgt No Wgt Wgt Wgt Wgt Wgt Wgt if Inc Same Dec tf Inc Same Dec theme Inc Same Dec context Inc Same Dec spec Inc Same Dec noise Inc Same Dec info_ Inc noise Same Dec Table 12: Effect of the aition of a characteristic to combinations of characteristics Bol figures inicate preominant effect of each characteristic The aition of context characteristic performe poorly in the smaller collections, performing more poorly when using weighting. In the larger collections the majority of combinations improve after the aition of context information. With exception of the CISI, the aition of the noise characteristic improves performance in both weighting an non-weighting conitions. The two ocument characteristics specificity an info_noise are very susceptible to how they are treate. The specificity characteristic tens to ecrease the effectiveness of a combination of characteristics if the 15

16 characteristics are not weighte. If the characteristics are weighte, then aition of specificity information is neutral: the combination performs as well as without the specificity information. The WSJ collection is the exception to this general conclusion. For this collection, uner no weighting, the aition of specificity increases the effectiveness of a combination. Uner weighting specificity ecreases the effectiveness of a combination. The info_noise characteristic tens to improve the effectiveness of a combination when using no weighting an to be neutral with respect to weighting, i.e. it oes not change the performance of the combination. The main exception to this is the MEDLARS collection in which info_noise tens to harm the performance of a combination when not using weighting. Having consiere which characteristics improve or worsene combinations, we now examine which combinations are affecte by the aition of new information. In Tables A.142 A.151, in the Appenix, we present a summary of how often iniviual characteristics will improve a combination containing another characteristic, e.g. how many combinations containing if are improve by the aition of tf. Uner both the weighting an non-weighting conitions the following generally hel: if improve combinations containing context more than other characteristics an improve combinations containing noise least of all tf improve combinations containing context or noise more than other characteristics an theme least theme improve combinations containing context most an combinations containing tf least context improve combinations containing noise least specificity improve combinations that containe theme an info_noise more than combinations containing other characteristics for the noise characteristic there were no general finings except that combinations containing if were usually less likely to be improve by the aition of noise information info_noise improve combinations containing theme an specificity most often. Weighting slightly altere which combinations performe well but the basic trens were the same across the conitions. On the larger collections, one effect of weighting was to reuce the effect of iniviual characteristics in that the effect of aing a characteristic was less likely to be epenent on which characteristics were alreay in the combination. One further observation is that term weighting schemes that represent similar features (e.g. if an noise which both represent global term statistics, an tf/theme which both represent within-ocument statistics) generally combine less well. That is combining these pairs of weights oes not generally help retrieval as much as combining complementary weights, e.g. if an tf, if an theme, etc. Combining the two ocument characteristics, however, oes seem to give better results. 6.4 Summary Our hypothesis was that combining evience combining characteristics of terms can improve retrieval effectiveness over retrieval by single characteristics. In section 6 we emonstrate that this was generally the case: all characteristics coul benefit from combination. Where combinations work well is where the aitional characteristics act as aitional means of ranking ocuments. That is separating ocuments by other sets of features. Other researchers have consiere this, e.g. Salton an Buckley examine if as a precision-enhancing funciton an tf as a recall-enhancing function, [SB88]. Similarly, Cooper, [Coo73] 17, iscusses the ifficulty of assessing likely utility without consiering aitional features of ocument content. However not all combinations are successful. Two aspects of combination that are likely to preict success are the nature of the characteristics complementary functions combine better an the success of the characteristic as a single retrieval function. Weighting the characteristics to reflect the strength of each characteristic as a single retrieval function is also generally a goo iea. However it can be ifficult to set optimal weights for two reasons: firstly it is likely that goo weights will be collection epenent as the iniviual characteristics have ifferent levels of effectiveness on ifferent collections. Seconly the weights shoul reflect the effectiveness of the characteristics relative to each other. However this becomes ifficult to assess when we combine characteristics, as we have to measure the relative strength of each 17 We are grateful to the anonymous referee for pointing us to this paper. 16

Sweden, The Baltic States and Poland November 2000

Sweden, The Baltic States and Poland November 2000 Folkbilning co-operation between Sween, The Baltic States an Polan 1990 2000 November 2000 TABLE OF CONTENTS FOREWORD...3 SUMMARY...4 I. CONCLUSIONS FROM THE COUNTRIES...6 I.1 Estonia...8 I.2 Latvia...12

More information

SANTIAGO CANYON COLLEGE Reading & English Placement Testing Information

SANTIAGO CANYON COLLEGE Reading & English Placement Testing Information SANTIAGO CANYON COLLEGE Reaing & English Placement Testing Information DO YOUR BEST on the Reaing & English Placement Test The Reaing & English placement test is esigne to assess stuents skills in reaing

More information

SPECIAL ARTICLES Pharmacy Education in Vietnam

SPECIAL ARTICLES Pharmacy Education in Vietnam American Journal of Pharmaceutical Eucation 2013; 77 (6) Article 114. SPECIAL ARTICLES Pharmacy Eucation in Vietnam Thi-Ha Vo, MSc, a,b Pierrick Beouch, PharmD, PhD, b,c Thi-Hoai Nguyen, PhD, a Thi-Lien-Huong

More information

The Good Judgment Project: A large scale test of different methods of combining expert predictions

The Good Judgment Project: A large scale test of different methods of combining expert predictions The Good Judgment Project: A large scale test of different methods of combining expert predictions Lyle Ungar, Barb Mellors, Jon Baron, Phil Tetlock, Jaime Ramos, Sam Swift The University of Pennsylvania

More information

Grade 6: Correlated to AGS Basic Math Skills

Grade 6: Correlated to AGS Basic Math Skills Grade 6: Correlated to AGS Basic Math Skills Grade 6: Standard 1 Number Sense Students compare and order positive and negative integers, decimals, fractions, and mixed numbers. They find multiples and

More information

SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF)

SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF) SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF) Hans Christian 1 ; Mikhael Pramodana Agus 2 ; Derwin Suhartono 3 1,2,3 Computer Science Department,

More information

A Case Study: News Classification Based on Term Frequency

A Case Study: News Classification Based on Term Frequency A Case Study: News Classification Based on Term Frequency Petr Kroha Faculty of Computer Science University of Technology 09107 Chemnitz Germany kroha@informatik.tu-chemnitz.de Ricardo Baeza-Yates Center

More information

On document relevance and lexical cohesion between query terms

On document relevance and lexical cohesion between query terms Information Processing and Management 42 (2006) 1230 1247 www.elsevier.com/locate/infoproman On document relevance and lexical cohesion between query terms Olga Vechtomova a, *, Murat Karamuftuoglu b,

More information

Page 1 of 11. Curriculum Map: Grade 4 Math Course: Math 4 Sub-topic: General. Grade(s): None specified

Page 1 of 11. Curriculum Map: Grade 4 Math Course: Math 4 Sub-topic: General. Grade(s): None specified Curriculum Map: Grade 4 Math Course: Math 4 Sub-topic: General Grade(s): None specified Unit: Creating a Community of Mathematical Thinkers Timeline: Week 1 The purpose of the Establishing a Community

More information

Cross Language Information Retrieval

Cross Language Information Retrieval Cross Language Information Retrieval RAFFAELLA BERNARDI UNIVERSITÀ DEGLI STUDI DI TRENTO P.ZZA VENEZIA, ROOM: 2.05, E-MAIL: BERNARDI@DISI.UNITN.IT Contents 1 Acknowledgment.............................................

More information

NCEO Technical Report 27

NCEO Technical Report 27 Home About Publications Special Topics Presentations State Policies Accommodations Bibliography Teleconferences Tools Related Sites Interpreting Trends in the Performance of Special Education Students

More information

Bridging Lexical Gaps between Queries and Questions on Large Online Q&A Collections with Compact Translation Models

Bridging Lexical Gaps between Queries and Questions on Large Online Q&A Collections with Compact Translation Models Bridging Lexical Gaps between Queries and Questions on Large Online Q&A Collections with Compact Translation Models Jung-Tae Lee and Sang-Bum Kim and Young-In Song and Hae-Chang Rim Dept. of Computer &

More information

South Carolina English Language Arts

South Carolina English Language Arts South Carolina English Language Arts A S O F J U N E 2 0, 2 0 1 0, T H I S S TAT E H A D A D O P T E D T H E CO M M O N CO R E S TAT E S TA N DA R D S. DOCUMENTS REVIEWED South Carolina Academic Content

More information

Extending Place Value with Whole Numbers to 1,000,000

Extending Place Value with Whole Numbers to 1,000,000 Grade 4 Mathematics, Quarter 1, Unit 1.1 Extending Place Value with Whole Numbers to 1,000,000 Overview Number of Instructional Days: 10 (1 day = 45 minutes) Content to Be Learned Recognize that a digit

More information

Rote rehearsal and spacing effects in the free recall of pure and mixed lists. By: Peter P.J.L. Verkoeijen and Peter F. Delaney

Rote rehearsal and spacing effects in the free recall of pure and mixed lists. By: Peter P.J.L. Verkoeijen and Peter F. Delaney Rote rehearsal and spacing effects in the free recall of pure and mixed lists By: Peter P.J.L. Verkoeijen and Peter F. Delaney Verkoeijen, P. P. J. L, & Delaney, P. F. (2008). Rote rehearsal and spacing

More information

Lecture 2: Quantifiers and Approximation

Lecture 2: Quantifiers and Approximation Lecture 2: Quantifiers and Approximation Case study: Most vs More than half Jakub Szymanik Outline Number Sense Approximate Number Sense Approximating most Superlative Meaning of most What About Counting?

More information

Linking Task: Identifying authors and book titles in verbose queries

Linking Task: Identifying authors and book titles in verbose queries Linking Task: Identifying authors and book titles in verbose queries Anaïs Ollagnier, Sébastien Fournier, and Patrice Bellot Aix-Marseille University, CNRS, ENSAM, University of Toulon, LSIS UMR 7296,

More information

Session 2B From understanding perspectives to informing public policy the potential and challenges for Q findings to inform survey design

Session 2B From understanding perspectives to informing public policy the potential and challenges for Q findings to inform survey design Session 2B From understanding perspectives to informing public policy the potential and challenges for Q findings to inform survey design Paper #3 Five Q-to-survey approaches: did they work? Job van Exel

More information

Student Morningness-Eveningness Type and Performance: Does Class Timing Matter?

Student Morningness-Eveningness Type and Performance: Does Class Timing Matter? Student Morningness-Eveningness Type and Performance: Does Class Timing Matter? Abstract Circadian rhythms have often been linked to people s performance outcomes, although this link has not been examined

More information

Pre-Algebra A. Syllabus. Course Overview. Course Goals. General Skills. Credit Value

Pre-Algebra A. Syllabus. Course Overview. Course Goals. General Skills. Credit Value Syllabus Pre-Algebra A Course Overview Pre-Algebra is a course designed to prepare you for future work in algebra. In Pre-Algebra, you will strengthen your knowledge of numbers as you look to transition

More information

THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS

THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS ELIZABETH ANNE SOMERS Spring 2011 A thesis submitted in partial

More information

Software Maintenance

Software Maintenance 1 What is Software Maintenance? Software Maintenance is a very broad activity that includes error corrections, enhancements of capabilities, deletion of obsolete capabilities, and optimization. 2 Categories

More information

Pedagogical Content Knowledge for Teaching Primary Mathematics: A Case Study of Two Teachers

Pedagogical Content Knowledge for Teaching Primary Mathematics: A Case Study of Two Teachers Pedagogical Content Knowledge for Teaching Primary Mathematics: A Case Study of Two Teachers Monica Baker University of Melbourne mbaker@huntingtower.vic.edu.au Helen Chick University of Melbourne h.chick@unimelb.edu.au

More information

Guide to the Uniform mark scale (UMS) Uniform marks in A-level and GCSE exams

Guide to the Uniform mark scale (UMS) Uniform marks in A-level and GCSE exams Guide to the Uniform mark scale (UMS) Uniform marks in A-level and GCSE exams This booklet explains why the Uniform mark scale (UMS) is necessary and how it works. It is intended for exams officers and

More information

Mathematics process categories

Mathematics process categories Mathematics process categories All of the UK curricula define multiple categories of mathematical proficiency that require students to be able to use and apply mathematics, beyond simple recall of facts

More information

Team Dispersal. Some shaping ideas

Team Dispersal. Some shaping ideas Team Dispersal Some shaping ideas The storyline is how distributed teams can be a liability or an asset or anything in between. It isn t simply a case of neutralizing the down side Nick Clare, January

More information

10.2. Behavior models

10.2. Behavior models User behavior research 10.2. Behavior models Overview Why do users seek information? How do they seek information? How do they search for information? How do they use libraries? These questions are addressed

More information

Build on students informal understanding of sharing and proportionality to develop initial fraction concepts.

Build on students informal understanding of sharing and proportionality to develop initial fraction concepts. Recommendation 1 Build on students informal understanding of sharing and proportionality to develop initial fraction concepts. Students come to kindergarten with a rudimentary understanding of basic fraction

More information

Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Probabilistic Latent Semantic Analysis Thomas Hofmann Presentation by Ioannis Pavlopoulos & Andreas Damianou for the course of Data Mining & Exploration 1 Outline Latent Semantic Analysis o Need o Overview

More information

Strategies for Solving Fraction Tasks and Their Link to Algebraic Thinking

Strategies for Solving Fraction Tasks and Their Link to Algebraic Thinking Strategies for Solving Fraction Tasks and Their Link to Algebraic Thinking Catherine Pearn The University of Melbourne Max Stephens The University of Melbourne

More information

Linking the Ohio State Assessments to NWEA MAP Growth Tests *

Linking the Ohio State Assessments to NWEA MAP Growth Tests * Linking the Ohio State Assessments to NWEA MAP Growth Tests * *As of June 2017 Measures of Academic Progress (MAP ) is known as MAP Growth. August 2016 Introduction Northwest Evaluation Association (NWEA

More information

Inquiry Learning Methodologies and the Disposition to Energy Systems Problem Solving

Inquiry Learning Methodologies and the Disposition to Energy Systems Problem Solving Inquiry Learning Methodologies and the Disposition to Energy Systems Problem Solving Minha R. Ha York University minhareo@yorku.ca Shinya Nagasaki McMaster University nagasas@mcmaster.ca Justin Riddoch

More information

Lecture 1: Machine Learning Basics

Lecture 1: Machine Learning Basics 1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3

More information

Rule Learning With Negation: Issues Regarding Effectiveness

Rule Learning With Negation: Issues Regarding Effectiveness Rule Learning With Negation: Issues Regarding Effectiveness S. Chua, F. Coenen, G. Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX Liverpool, United

More information

Radius STEM Readiness TM

Radius STEM Readiness TM Curriculum Guide Radius STEM Readiness TM While today s teens are surrounded by technology, we face a stark and imminent shortage of graduates pursuing careers in Science, Technology, Engineering, and

More information

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS 1 CALIFORNIA CONTENT STANDARDS: Chapter 1 ALGEBRA AND WHOLE NUMBERS Algebra and Functions 1.4 Students use algebraic

More information

Numeracy Medium term plan: Summer Term Level 2C/2B Year 2 Level 2A/3C

Numeracy Medium term plan: Summer Term Level 2C/2B Year 2 Level 2A/3C Numeracy Medium term plan: Summer Term Level 2C/2B Year 2 Level 2A/3C Using and applying mathematics objectives (Problem solving, Communicating and Reasoning) Select the maths to use in some classroom

More information

Term Weighting based on Document Revision History

Term Weighting based on Document Revision History Term Weighting based on Document Revision History Sérgio Nunes, Cristina Ribeiro, and Gabriel David INESC Porto, DEI, Faculdade de Engenharia, Universidade do Porto. Rua Dr. Roberto Frias, s/n. 4200-465

More information

Writing a composition

Writing a composition A good composition has three elements: Writing a composition an introduction: A topic sentence which contains the main idea of the paragraph. a body : Supporting sentences that develop the main idea. a

More information

CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2

CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2 1 CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2 Peter A. Chew, Brett W. Bader, Ahmed Abdelali Proceedings of the 13 th SIGKDD, 2007 Tiago Luís Outline 2 Cross-Language IR (CLIR) Latent Semantic Analysis

More information

Assignment 1: Predicting Amazon Review Ratings

Assignment 1: Predicting Amazon Review Ratings Assignment 1: Predicting Amazon Review Ratings 1 Dataset Analysis Richard Park r2park@acsmail.ucsd.edu February 23, 2015 The dataset selected for this assignment comes from the set of Amazon reviews for

More information

Mathematics Scoring Guide for Sample Test 2005

Mathematics Scoring Guide for Sample Test 2005 Mathematics Scoring Guide for Sample Test 2005 Grade 4 Contents Strand and Performance Indicator Map with Answer Key...................... 2 Holistic Rubrics.......................................................

More information

Research Update. Educational Migration and Non-return in Northern Ireland May 2008

Research Update. Educational Migration and Non-return in Northern Ireland May 2008 Research Update Educational Migration and Non-return in Northern Ireland May 2008 The Equality Commission for Northern Ireland (hereafter the Commission ) in 2007 contracted the Employment Research Institute

More information

UMass at TDT Similarity functions 1. BASIC SYSTEM Detection algorithms. set globally and apply to all clusters.

UMass at TDT Similarity functions 1. BASIC SYSTEM Detection algorithms. set globally and apply to all clusters. UMass at TDT James Allan, Victor Lavrenko, David Frey, and Vikas Khandelwal Center for Intelligent Information Retrieval Department of Computer Science University of Massachusetts Amherst, MA 3 We spent

More information

1 3-5 = Subtraction - a binary operation

1 3-5 = Subtraction - a binary operation High School StuDEnts ConcEPtions of the Minus Sign Lisa L. Lamb, Jessica Pierson Bishop, and Randolph A. Philipp, Bonnie P Schappelle, Ian Whitacre, and Mindy Lewis - describe their research with students

More information

Progress Monitoring for Behavior: Data Collection Methods & Procedures

Progress Monitoring for Behavior: Data Collection Methods & Procedures Progress Monitoring for Behavior: Data Collection Methods & Procedures This event is being funded with State and/or Federal funds and is being provided for employees of school districts, employees of the

More information

have to be modeled) or isolated words. Output of the system is a grapheme-tophoneme conversion system which takes as its input the spelling of words,

have to be modeled) or isolated words. Output of the system is a grapheme-tophoneme conversion system which takes as its input the spelling of words, A Language-Independent, Data-Oriented Architecture for Grapheme-to-Phoneme Conversion Walter Daelemans and Antal van den Bosch Proceedings ESCA-IEEE speech synthesis conference, New York, September 1994

More information

CS Machine Learning

CS Machine Learning CS 478 - Machine Learning Projects Data Representation Basic testing and evaluation schemes CS 478 Data and Testing 1 Programming Issues l Program in any platform you want l Realize that you will be doing

More information

Artificial Neural Networks written examination

Artificial Neural Networks written examination 1 (8) Institutionen för informationsteknologi Olle Gällmo Universitetsadjunkt Adress: Lägerhyddsvägen 2 Box 337 751 05 Uppsala Artificial Neural Networks written examination Monday, May 15, 2006 9 00-14

More information

Speech Recognition at ICSI: Broadcast News and beyond

Speech Recognition at ICSI: Broadcast News and beyond Speech Recognition at ICSI: Broadcast News and beyond Dan Ellis International Computer Science Institute, Berkeley CA Outline 1 2 3 The DARPA Broadcast News task Aspects of ICSI

More information

Massachusetts Department of Elementary and Secondary Education. Title I Comparability

Massachusetts Department of Elementary and Secondary Education. Title I Comparability Massachusetts Department of Elementary and Secondary Education Title I Comparability 2009-2010 Title I provides federal financial assistance to school districts to provide supplemental educational services

More information

RCPCH MMC Cohort Study (Part 4) March 2016

RCPCH MMC Cohort Study (Part 4) March 2016 RCPCH MMC Cohort Study (Part 4) March 2016 Acknowledgements Dr Simon Clark, Officer for Workforce Planning, RCPCH Dr Carol Ewing, Vice President Health Services, RCPCH Dr Daniel Lumsden, Former Chair,

More information

DMA CLUSTER CALCULATIONS POLICY

DMA CLUSTER CALCULATIONS POLICY DMA CLUSTER CALCULATIONS POLICY Watlington C P School Shouldham Windows User HEWLETT-PACKARD [Company address] Riverside Federation CONTENTS Titles Page Schools involved 2 Rationale 3 Aims and principles

More information

Longitudinal Analysis of the Effectiveness of DCPS Teachers

Longitudinal Analysis of the Effectiveness of DCPS Teachers F I N A L R E P O R T Longitudinal Analysis of the Effectiveness of DCPS Teachers July 8, 2014 Elias Walsh Dallas Dotter Submitted to: DC Education Consortium for Research and Evaluation School of Education

More information

Functional Skills Mathematics Level 2 assessment

Functional Skills Mathematics Level 2 assessment Functional Skills Mathematics Level 2 assessment www.cityandguilds.com September 2015 Version 1.0 Marking scheme ONLINE V2 Level 2 Sample Paper 4 Mark Represent Analyse Interpret Open Fixed S1Q1 3 3 0

More information

How to Judge the Quality of an Objective Classroom Test

How to Judge the Quality of an Objective Classroom Test How to Judge the Quality of an Objective Classroom Test Technical Bulletin #6 Evaluation and Examination Service The University of Iowa (319) 335-0356 HOW TO JUDGE THE QUALITY OF AN OBJECTIVE CLASSROOM

More information

On the Combined Behavior of Autonomous Resource Management Agents

On the Combined Behavior of Autonomous Resource Management Agents On the Combined Behavior of Autonomous Resource Management Agents Siri Fagernes 1 and Alva L. Couch 2 1 Faculty of Engineering Oslo University College Oslo, Norway siri.fagernes@iu.hio.no 2 Computer Science

More information

Let s think about how to multiply and divide fractions by fractions!

Let s think about how to multiply and divide fractions by fractions! Let s think about how to multiply and divide fractions by fractions! June 25, 2007 (Monday) Takehaya Attached Elementary School, Tokyo Gakugei University Grade 6, Class # 1 (21 boys, 20 girls) Instructor:

More information

4.0 CAPACITY AND UTILIZATION

4.0 CAPACITY AND UTILIZATION 4.0 CAPACITY AND UTILIZATION The capacity of a school building is driven by four main factors: (1) the physical size of the instructional spaces, (2) the class size limits, (3) the schedule of uses, and

More information

Online Updating of Word Representations for Part-of-Speech Tagging

Online Updating of Word Representations for Part-of-Speech Tagging Online Updating of Word Representations for Part-of-Speech Tagging Wenpeng Yin LMU Munich wenpeng@cis.lmu.de Tobias Schnabel Cornell University tbs49@cornell.edu Hinrich Schütze LMU Munich inquiries@cislmu.org

More information

Cal s Dinner Card Deals

Cal s Dinner Card Deals Cal s Dinner Card Deals Overview: In this lesson students compare three linear functions in the context of Dinner Card Deals. Students are required to interpret a graph for each Dinner Card Deal to help

More information

Grade 5 + DIGITAL. EL Strategies. DOK 1-4 RTI Tiers 1-3. Flexible Supplemental K-8 ELA & Math Online & Print

Grade 5 + DIGITAL. EL Strategies. DOK 1-4 RTI Tiers 1-3. Flexible Supplemental K-8 ELA & Math Online & Print Standards PLUS Flexible Supplemental K-8 ELA & Math Online & Print Grade 5 SAMPLER Mathematics EL Strategies DOK 1-4 RTI Tiers 1-3 15-20 Minute Lessons Assessments Consistent with CA Testing Technology

More information

Unequal Opportunity in Environmental Education: Environmental Education Programs and Funding at Contra Costa Secondary Schools.

Unequal Opportunity in Environmental Education: Environmental Education Programs and Funding at Contra Costa Secondary Schools. Unequal Opportunity in Environmental Education: Environmental Education Programs and Funding at Contra Costa Secondary Schools Angela Freitas Abstract Unequal opportunity in education threatens to deprive

More information

BENCHMARK TREND COMPARISON REPORT:

BENCHMARK TREND COMPARISON REPORT: National Survey of Student Engagement (NSSE) BENCHMARK TREND COMPARISON REPORT: CARNEGIE PEER INSTITUTIONS, 2003-2011 PREPARED BY: ANGEL A. SANCHEZ, DIRECTOR KELLI PAYNE, ADMINISTRATIVE ANALYST/ SPECIALIST

More information

Stacks Teacher notes. Activity description. Suitability. Time. AMP resources. Equipment. Key mathematical language. Key processes

Stacks Teacher notes. Activity description. Suitability. Time. AMP resources. Equipment. Key mathematical language. Key processes Stacks Teacher notes Activity description (Interactive not shown on this sheet.) Pupils start by exploring the patterns generated by moving counters between two stacks according to a fixed rule, doubling

More information

EDUCATIONAL ATTAINMENT

EDUCATIONAL ATTAINMENT EDUCATIONAL ATTAINMENT By 2030, at least 60 percent of Texans ages 25 to 34 will have a postsecondary credential or degree. Target: Increase the percent of Texans ages 25 to 34 with a postsecondary credential.

More information

Loughton School s curriculum evening. 28 th February 2017

Loughton School s curriculum evening. 28 th February 2017 Loughton School s curriculum evening 28 th February 2017 Aims of this session Share our approach to teaching writing, reading, SPaG and maths. Share resources, ideas and strategies to support children's

More information

Cross-lingual Text Fragment Alignment using Divergence from Randomness

Cross-lingual Text Fragment Alignment using Divergence from Randomness Cross-lingual Text Fragment Alignment using Divergence from Randomness Sirvan Yahyaei, Marco Bonzanini, and Thomas Roelleke Queen Mary, University of London Mile End Road, E1 4NS London, UK {sirvan,marcob,thor}@eecs.qmul.ac.uk

More information

Learning From the Past with Experiment Databases

Learning From the Past with Experiment Databases Learning From the Past with Experiment Databases Joaquin Vanschoren 1, Bernhard Pfahringer 2, and Geoff Holmes 2 1 Computer Science Dept., K.U.Leuven, Leuven, Belgium 2 Computer Science Dept., University

More information

Grade Dropping, Strategic Behavior, and Student Satisficing

Grade Dropping, Strategic Behavior, and Student Satisficing Grade Dropping, Strategic Behavior, and Student Satisficing Lester Hadsell Department of Economics State University of New York, College at Oneonta Oneonta, NY 13820 hadsell@oneonta.edu Raymond MacDermott

More information

What s in a Step? Toward General, Abstract Representations of Tutoring System Log Data

What s in a Step? Toward General, Abstract Representations of Tutoring System Log Data What s in a Step? Toward General, Abstract Representations of Tutoring System Log Data Kurt VanLehn 1, Kenneth R. Koedinger 2, Alida Skogsholm 2, Adaeze Nwaigwe 2, Robert G.M. Hausmann 1, Anders Weinstein

More information

Financing Education In Minnesota

Financing Education In Minnesota Financing Education In Minnesota 2016-2017 Created with Tagul.com A Publication of the Minnesota House of Representatives Fiscal Analysis Department August 2016 Financing Education in Minnesota 2016-17

More information

GCSE Mathematics B (Linear) Mark Scheme for November Component J567/04: Mathematics Paper 4 (Higher) General Certificate of Secondary Education

GCSE Mathematics B (Linear) Mark Scheme for November Component J567/04: Mathematics Paper 4 (Higher) General Certificate of Secondary Education GCSE Mathematics B (Linear) Component J567/04: Mathematics Paper 4 (Higher) General Certificate of Secondary Education Mark Scheme for November 2014 Oxford Cambridge and RSA Examinations OCR (Oxford Cambridge

More information

Algebra 1 Summer Packet

Algebra 1 Summer Packet Algebra 1 Summer Packet Name: Solve each problem and place the answer on the line to the left of the problem. Adding Integers A. Steps if both numbers are positive. Example: 3 + 4 Step 1: Add the two numbers.

More information

TRAVEL TIME REPORT. Casualty Actuarial Society Education Policy Committee October 2001

TRAVEL TIME REPORT. Casualty Actuarial Society Education Policy Committee October 2001 TRAVEL TIME REPORT Casualty Actuarial Society Education Policy Committee October 2001 The Education Policy Committee has completed its annual review of travel time. As was the case last year, we do expect

More information

The Role of String Similarity Metrics in Ontology Alignment

The Role of String Similarity Metrics in Ontology Alignment The Role of String Similarity Metrics in Ontology Alignment Michelle Cheatham and Pascal Hitzler August 9, 2013 1 Introduction Tim Berners-Lee originally envisioned a much different world wide web than

More information

GCSE. Mathematics A. Mark Scheme for January General Certificate of Secondary Education Unit A503/01: Mathematics C (Foundation Tier)

GCSE. Mathematics A. Mark Scheme for January General Certificate of Secondary Education Unit A503/01: Mathematics C (Foundation Tier) GCSE Mathematics A General Certificate of Secondary Education Unit A503/0: Mathematics C (Foundation Tier) Mark Scheme for January 203 Oxford Cambridge and RSA Examinations OCR (Oxford Cambridge and RSA)

More information

Diagnostic Test. Middle School Mathematics

Diagnostic Test. Middle School Mathematics Diagnostic Test Middle School Mathematics Copyright 2010 XAMonline, Inc. All rights reserved. No part of the material protected by this copyright notice may be reproduced or utilized in any form or by

More information

OCR for Arabic using SIFT Descriptors With Online Failure Prediction

OCR for Arabic using SIFT Descriptors With Online Failure Prediction OCR for Arabic using SIFT Descriptors With Online Failure Prediction Andrey Stolyarenko, Nachum Dershowitz The Blavatnik School of Computer Science Tel Aviv University Tel Aviv, Israel Email: stloyare@tau.ac.il,

More information

Individual Component Checklist L I S T E N I N G. for use with ONE task ENGLISH VERSION

Individual Component Checklist L I S T E N I N G. for use with ONE task ENGLISH VERSION L I S T E N I N G Individual Component Checklist for use with ONE task ENGLISH VERSION INTRODUCTION This checklist has been designed for use as a practical tool for describing ONE TASK in a test of listening.

More information

Learning to Rank with Selection Bias in Personal Search

Learning to Rank with Selection Bias in Personal Search Learning to Rank with Selection Bias in Personal Search Xuanhui Wang, Michael Bendersky, Donald Metzler, Marc Najork Google Inc. Mountain View, CA 94043 {xuanhui, bemike, metzler, najork}@google.com ABSTRACT

More information

Chapter 10 APPLYING TOPIC MODELING TO FORENSIC DATA. 1. Introduction. Alta de Waal, Jacobus Venter and Etienne Barnard

Chapter 10 APPLYING TOPIC MODELING TO FORENSIC DATA. 1. Introduction. Alta de Waal, Jacobus Venter and Etienne Barnard Chapter 10 APPLYING TOPIC MODELING TO FORENSIC DATA Alta de Waal, Jacobus Venter and Etienne Barnard Abstract Most actionable evidence is identified during the analysis phase of digital forensic investigations.

More information

The stages of event extraction

The stages of event extraction The stages of event extraction David Ahn Intelligent Systems Lab Amsterdam University of Amsterdam ahn@science.uva.nl Abstract Event detection and recognition is a complex task consisting of multiple sub-tasks

More information

Calculators in a Middle School Mathematics Classroom: Helpful or Harmful?

Calculators in a Middle School Mathematics Classroom: Helpful or Harmful? University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Action Research Projects Math in the Middle Institute Partnership 7-2008 Calculators in a Middle School Mathematics Classroom:

More information

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur Module 12 Machine Learning 12.1 Instructional Objective The students should understand the concept of learning systems Students should learn about different aspects of a learning system Students should

More information

Backwards Numbers: A Study of Place Value. Catherine Perez

Backwards Numbers: A Study of Place Value. Catherine Perez Backwards Numbers: A Study of Place Value Catherine Perez Introduction I was reaching for my daily math sheet that my school has elected to use and in big bold letters in a box it said: TO ADD NUMBERS

More information

On-the-Fly Customization of Automated Essay Scoring

On-the-Fly Customization of Automated Essay Scoring Research Report On-the-Fly Customization of Automated Essay Scoring Yigal Attali Research & Development December 2007 RR-07-42 On-the-Fly Customization of Automated Essay Scoring Yigal Attali ETS, Princeton,

More information

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS

OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS Václav Kocian, Eva Volná, Michal Janošek, Martin Kotyrba University of Ostrava Department of Informatics and Computers Dvořákova 7,

More information

University of Exeter College of Humanities. Assessment Procedures 2010/11

University of Exeter College of Humanities. Assessment Procedures 2010/11 University of Exeter College of Humanities Assessment Procedures 2010/11 This document describes the conventions and procedures used to assess, progress and classify UG students within the College of Humanities.

More information

Evaluation of Learning Management System software. Part II of LMS Evaluation

Evaluation of Learning Management System software. Part II of LMS Evaluation Version DRAFT 1.0 Evaluation of Learning Management System software Author: Richard Wyles Date: 1 August 2003 Part II of LMS Evaluation Open Source e-learning Environment and Community Platform Project

More information

South Carolina College- and Career-Ready Standards for Mathematics. Standards Unpacking Documents Grade 5

South Carolina College- and Career-Ready Standards for Mathematics. Standards Unpacking Documents Grade 5 South Carolina College- and Career-Ready Standards for Mathematics Standards Unpacking Documents Grade 5 South Carolina College- and Career-Ready Standards for Mathematics Standards Unpacking Documents

More information

Focus of the Unit: Much of this unit focuses on extending previous skills of multiplication and division to multi-digit whole numbers.

Focus of the Unit: Much of this unit focuses on extending previous skills of multiplication and division to multi-digit whole numbers. Approximate Time Frame: 3-4 weeks Connections to Previous Learning: In fourth grade, students fluently multiply (4-digit by 1-digit, 2-digit by 2-digit) and divide (4-digit by 1-digit) using strategies

More information

Activities, Exercises, Assignments Copyright 2009 Cem Kaner 1

Activities, Exercises, Assignments Copyright 2009 Cem Kaner 1 Patterns of activities, iti exercises and assignments Workshop on Teaching Software Testing January 31, 2009 Cem Kaner, J.D., Ph.D. kaner@kaner.com Professor of Software Engineering Florida Institute of

More information

School Competition and Efficiency with Publicly Funded Catholic Schools David Card, Martin D. Dooley, and A. Abigail Payne

School Competition and Efficiency with Publicly Funded Catholic Schools David Card, Martin D. Dooley, and A. Abigail Payne School Competition and Efficiency with Publicly Funded Catholic Schools David Card, Martin D. Dooley, and A. Abigail Payne Web Appendix See paper for references to Appendix Appendix 1: Multiple Schools

More information

The Role of the Head in the Interpretation of English Deverbal Compounds

The Role of the Head in the Interpretation of English Deverbal Compounds The Role of the Head in the Interpretation of English Deverbal Compounds Gianina Iordăchioaia i, Lonneke van der Plas ii, Glorianna Jagfeld i (Universität Stuttgart i, University of Malta ii ) Wen wurmt

More information

A Note on Structuring Employability Skills for Accounting Students

A Note on Structuring Employability Skills for Accounting Students A Note on Structuring Employability Skills for Accounting Students Jon Warwick and Anna Howard School of Business, London South Bank University Correspondence Address Jon Warwick, School of Business, London

More information

Vocabulary Agreement Among Model Summaries And Source Documents 1

Vocabulary Agreement Among Model Summaries And Source Documents 1 Vocabulary Agreement Among Model Summaries And Source Documents 1 Terry COPECK, Stan SZPAKOWICZ School of Information Technology and Engineering University of Ottawa 800 King Edward Avenue, P.O. Box 450

More information

Rule Learning with Negation: Issues Regarding Effectiveness

Rule Learning with Negation: Issues Regarding Effectiveness Rule Learning with Negation: Issues Regarding Effectiveness Stephanie Chua, Frans Coenen, and Grant Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX

More information

Engineers and Engineering Brand Monitor 2015

Engineers and Engineering Brand Monitor 2015 Engineers and Engineering Brand Monitor 2015 Key Findings Prepared for Engineering UK By IFF Research 7 September 2015 We gratefully acknowledge the support of Pearson in delivering this study Contact

More information

Running head: DELAY AND PROSPECTIVE MEMORY 1

Running head: DELAY AND PROSPECTIVE MEMORY 1 Running head: DELAY AND PROSPECTIVE MEMORY 1 In Press at Memory & Cognition Effects of Delay of Prospective Memory Cues in an Ongoing Task on Prospective Memory Task Performance Dawn M. McBride, Jaclyn

More information