A metric for automatically evaluating coherent summaries via context chains

Size: px
Start display at page:

Download "A metric for automatically evaluating coherent summaries via context chains"

Transcription

1 2009 IEEE International Conference on Semantic Computing A metric for automatically evaluating coherent summaries via context chains Frank Schilder and Ravi Kondadadi Thomson Reuters Corporation Research & Development 610 Opperman Drive St. Paul, MN 55104, USA frank.schilder,ravikumar.kondadadi@thomsonreuters.com Abstract This paper introduces a new metric for automatically evaluation summaries called ContextChain. Based on an in-depth analysis of the TAC 2008 update summarization results, we show that previous automatic metrics such as ROUGE-2 and BE cannot reliably predict strong performing systems. We introduce two new terms called Correlation Recall and Correlation Precision and discuss how they cast more light on the coverage and the correctness of the respective metric. Our newly proposed metric called ContextChain incorporates findings from Giannakopoulos et al. (2008) and Barzilay and Lapata (2008) [2]. We show that our metric correlates with responsiveness scores even for the top n systems that participated in the TAC 2008 update summarization task, whereas ROUGE-2 and BE do not show a correlation for the top 25 systems. 1. Introduction NIST has been organizing summarization competitions for the last several years and produced manually evaluations based on a metric called Responsiveness. Responsiveness is defined as a metric for how well a summary can meet the information need of a user asking a complex question. Creating such an evaluation is very labor-intensive, because every summary has to be judged by a human. Consequently, much effort has been put into the development of an automatic metric for evaluating summarization systems in order to advance the state-of-the-art for automatic summarization more quickly. NIST evaluated automatically generated summaries by utilizing the ROUGE metric for the recent Document Understanding Conferences (DUC) [9] and last year s Text Analysis Conference (TAC) [3]. ROUGE relies on the statistical analysis of co-occurring word n-grams between the peer and reference summary. However, the last two summarization tasks defined by NIST for 2007 and 2008 showed that ROUGE has two shortcomings. First, the best systems for the DUC 2007 competition received ROUGE-2 values close or equivalent to some human written summaries. Given this situation, it becomes more and more difficult to measure progress via an automatic metric. Second, a closer analysis of the top systems showed that there was no or very little correlation between the automatic metric and the Responsiveness score, although the overall correlation between these automatic metrics and responsiveness was still high for the full set of 58 evaluated systems [10]. We conclude that low ROUGE-2 (and BE) scores can be seen as reliable indication for low summarization performance, but high ROUGE-2 (and BE) scores are not a sufficient differentiator for good and very good performing systems. The two main contributions of our paper are the following 1. a more detailed analysis of how automatic metrics indicate top performing systems than described in [10]. We present a comparison of the top systems sorted according to (a) Responsiveness and (b) the automatic metric. We note that an analysis according to the first sorting criteria indicates whether all good systems are reliably found by the automatic metric, whereas an analysis according to sorting criteria (b) describes to what degree the metric delivers correct results. These two views can be seen as recall and precision, respectively. 2. a new evaluation metric called ContextChain focussing on the local coherence of the automatic summaries. The new evaluation metric we propose relies on an n- gram graph and puts more emphasis on the linear coherence of the written summaries. An automatically generated summary may receive a high ROUGE score, because it contains many relevant n-grams, but may be /09 $ IEEE DOI /ICSC

2 badly structured because the system did not consider the local coherence constraints. Our automatic metric tries to capture the local coherence by extracting the local context of named entities and keeping the typical sequence of how entities and concepts are introduced. Our analysis shows that ContextChain offers significantly better correlation between responsiveness than previously used automatic metrics for the top performing systems and comparable performance and partly better performance than the recently proposed AutoSumm metric [4]. The remainder of this paper is organized as follows. First, we discuss related work before we describe in more detail our new approach. Section 3 provides an overview of our new evaluation metric. Section 4 gives the main task definition for TAC Section 5 discusses the evaluation for the TAC 2008 update task while a more in-depth analysis of this evaluation and more automatic metrics is presented in section 6. Section 7 concludes the paper and discusses next research steps. 2. Related Work ROUGE [6, 7] is one of the first automatic summarization evaluation metrics proposed. ROUGE uses lexical n-grams to compare human written model summaries with automatically generated summaries. Hovy et. al. Later, [5] proposed an approach to automatic evaluation based on the concept of Basic Element. A Basic Element (BE) is a semantic unit extracted from a sentence such as subject-object relation, modifier-object relation. Systems with higher overlap of system-summary BEs and humansummary BEs get higher BE scores. Recently, AutoSummENG was introduced as a summarization evaluation method that evaluates summaries by extracting and comparing graphs of character and word n- grams [4]. Both the model and system summaries are represented as graphs. Edges in the graph are created based on the adjacency relation between n-grams. The edges are weighted according to the distance between the neighbors or the number of cooccurences with in the text. Similarity between two graphs is computed as number of common edges. Similarity can also include the weights of the common edges. In section 6, we evaluate these three automatic metrics with the TAC 2008 evaluation results. Two other proposals for new evaluation metrics address the question of improving the evalution metric in general, but they do not address the problem of low correlations for top n system discussed by this paper. Tratz and Hovy (2008) [11] describe a new implementation of the BE method, called BE with Transformations for Evaluation (BEwTE) that includes a significantly improved matching capability using a variety of operations to transform and match BEs in various ways. Louis and Nenkova (2008) [1] use features based on distribution of terms in the input summary and the model summary. They use KL, JS Divergence and cosine similarity to compute the similarity of term distribution of the input and the model summary. 3. Context Chains and n-gram graphs [4] propose a method called AutoSummENG that generate n-gram graphs for the model summaries and the automatically generated summaries. The AutoSummENG summarization evaluation metric is based on the similarity between the n-gram graph representations for the generated system summaries and model summaries. An n-gram graph can be generated for word or character windows. An 2-gram graph for n=2 for the following sentence can constructed by first generating all 2-grams: A quick brown fox jumps over the lazy dog. Figure 1 shows the complete graph generated from this sentence. In addition, weights on the edges can indidcate the distance between the neighbors or the number of occurrences in the text. By creating edges between the adjacent n-grams, this approach takes the contextual information into consideration as opposed to approaches that only use the n-gram overlap between the system and model summaries. Similarity between the graphs is computed via the Value Similarity, the Size Similarity, and the Co-occurrence Similarity. 1 They show that their approach is superior over past automatic metrics such as ROUGE and BE for the DUC 2005, 2006 and 2007 summarization tasks. Our approach is an extension of AutoSummENG that generates n-gram graphs based on co-reference chains. Our approach also models local coherence by establishing chains of potentially co-referent named entities and definite descriptions. The n-gram graph is then generated from the context of these referents. Consider the the beginning of a news story shown in figure 2. These n-grams can be seen as the events the entities mentioned in the summaries are involved in and the links determine the sequence in which the events should be mentioned. The links, therefore, capture the local coherence, as found in the model summaries. Note that this is a main difference between our approach and the other purely n-gram based approaches. An automatically generated summary may share lots of n-grams with the model summaries, but the sequence of how the events are presented may be incoherent and hence decreases the readability of the summary. 1 See [4] on how to compute these scores. 66

3 Figure 1. An n-gram graph The Justice Department is conducting an anti-trust trial against Microsoft Corp with evidence that the company is increasingly attempting to crush competitors. Microsoft is accused of trying to forcefully buy into markets... All context 4-grams (minus stop words) for the named entity Microsoft: Department conducting anti-trust trial evidence company increasingly attempting accused trying forcefully buy Two context chains are generated: Department conducting anti-trust trial accused trying forcefully buy evidence company increasingly attempting accused trying forcefully buy Figure 2. Example text and 2 example context chains generated for one named entity We implemented our approach within the AutoSumm GUI that is freely available. For the named entity extraction and chunking, we used LingPipe s named entity tagger and chunker TAC 2008 main task descriptions The main task in 2008 addressed the challenge of providing an update summary for a cluster of documents, given that the user has already read documents on this topic. Consequently, the update summary should not contain information that the user is already aware of. More precisely, the task is divided into two sub-tasks. The goal of the first summarization sub-task is to produce a normal query-based multi-document summary of a cluster of news documents. The second sub-task assumes that the information described in the first cluster is already known to a user who would like to receive a summary for a second cluster. The first cluster of documents needs to be summarized as a multi-document summary, whereas the second cluster is to be summarized taking into account the knowledge present in the information described by the first cluster. The input for this entire update task is a list of topics, each of which contain a title, a sequence of questions and 2 Baldwin, B. and B. Carpenter. LingPipe. two clusters of 10 documents each. An example topic and the question is the following: <title> Kyoto Protocol Implementation </title> <narrative> Track the implementation of key elements of the Kyoto Protocol by the 30 signatory countries. Describe what specific measures will actually be taken or not taken by various countries in response to the key climate change mitigation elements of the Kyoto Protocol. </narrative> 5. TAC 2008 Evaluations NIST carried out a manual evaluation and an automatic evaluation. This section presents the overall results with respect to how different metrics correlate to each other For more detailed information on the systems approaches and performances see [3] 67

4 5.1. Manual evaluation Summaries were manually evaluated for linguistic quality, Responsiveness and Pyramid score. The overall Responsiveness score is an integer between 1 (very poor) and 5 (very good) and is based on both the linguistic quality of the summary and the amount of information in the summary that helps to satisfy the information need expressed in the topic narrative. The Pyramid scores were created by NIST assessors from the four model summaries for each document set and the peer summaries using the pyramid guidelines provided by Columbia University. Responsiveness and Pyramid scores correlate highly with each other, as shown in Figure 3. The Pearson coefficient for these two manual metrics was Responsiveness Update task: manual evaluations Pyramid Figure 3. Responsiveness/Pyramid 5.2. Automatic evaluation At TAC 2008, two automatic metrics were used: ROUGE and BE. Analyzing the correlation between the ROUGE-2 and Responsiveness score, two observations can be made. The Pearson coefficient for all systems is still high, but not as high as for Responsiveness vs. Pyramid score (i.e., ). However, focussing only on the top systems there is no correlation between Responsiveness and ROUGE-2. For example, figure 4 indicates that the top 22 systems according to ROUGE-2 (i.e., ROUGE-2 scores > 0.08) show no correlation between ROUGE-2 and Responsiveness. The Pearson coefficient for these systems is Given this first observation, we may conclude that only low ROUGE-2 scores (i.e., < 0.08) can be seen as indication for the summarization performance, but high ROUGE- 2 scores cannot differentiate good and very good performing systems. Responsiveness top rest top Update task: manual vs. automatic metrics ROUGE 2 Figure 4. Responsiveness/ROUGE-2 For the BE evaluation, a similar picture emerged. The correlation between Responsiveness and BE for all systems was relatively high. Pearson s r was Considering only the top 22 systems, one observes that the Pearson coefficient for BE and Responsiveness showed a weak correlation between the automatic and the manual evaluation metric. The Pearson coefficient is and the confidence intervals show that this correlation is not significant. 6. More automatic metrics Given the low correlations between ROUGE-2 and BE for the top 22 systems, we investigated other metrics that may show a higher correlation. We tried the AutoSumm metric and developed our own metric utilizing the AutoSumm software. In our experiments, we ran the TAC 2008 systems through the AutoSumm and our new system called ContextChain for different n top systems (n = 10, 15, 20, 25, 30, 35). For the obtained results, we computed Pearson coefficients in two ways: Responsiveness-sorted: The two vectors of results were sorted according to Responsiveness scores. This could mean that systems that obtained high scores from the automatic metric, but low Responsiveness 68

5 Pearson's r coefficient scores were not considered for the correlation evaluation. Automatic evaluation metric-sorted: the two vectors of results were sorted according to the automatic metric. This could mean that systems that obtained high Responsiveness scores, but low automatic metric scores were not considered for the correlation evaluation Correlation Recall n top systems Metrics ContextChain AutoSummENG (words) BE AutoSummENG (characters) ROUGE-2 Figure 5. Correlations for n top systems sorted by Responsiveness Figure 5 shows the Pearson coefficients for the top systems, respectively, if sorted according to Responsiveness. This set-up of the experiment focusses on the top n systems determined by the manual evaluation metric. An automatic metric that shows high coefficients throughout the different number of top systems, shows high coverage (or recall) of the top performing systems. We define this set-up as Correlation Recall. Conversely, an automatic system that shows a consistently high coefficient for systems sorted according to the automatic metric, is reliable in terms of its precision. In other words, a high automatic score is likely to indicate a high performing system in terms of Responsiveness. We define this set-up as Correlation Precision. Figure 5 contains the Pearson coefficients for the top n systems for all metrics dicussed. The values for ROUGE-2 and BE are generally low and only high n allow the conclusion that the metric correlates with the human evaluation metric (cf. tables in appendix). Note that AutoSummENG using characters for the n-gram graphs does not preform Pearson's r coefficient Correlation Precision n top systems Metrics ContextChain AutoSummENG (words) BE AutoSummENG (characters) ROUGE-2 Figure 6. Correlations for n top systems sorted by evaluation metric very well either. We tried out AutoSummENG with words instead which resulted in a better overall performance similar to our proposed ContectChain metric. The Correlation Precision, on the other hand, seems to be a better indicator for how good the metric can predict strong performing systems. Figure 6 indicates that the previously used automatic metrics ROUGE-2 and BE show low or no correlations for up to the 25 top systems. In fact, for the top 10 or 15 systems the relation between Responsiveness and ROUGE-2 may even be inverse. 4 BE shows a better performance than ROUGE in that respect, but its Pearson coefficients are still lower than the other two metrics. Auto- SummENG based on characters performs again not as well as when words are used for generating the graph. ContextChain is very similar to AutoSummENG with words while showing a higher coefficient when only the top 10 systems are considered. This difference, however, is not significant. We also analyzed the results of these experiments regarding their significance. The appendix contains tables showing the confidence intervals computed via the Fishers r-to-z transformation [8] for Correlation Recall and Correlation Precision, respectively. 4 Bear in mind that the confidence intervals for these small samples are very large. 69

6 7. Conclusions This paper introduced a new metric for automatically evaluation summaries. Based on an in-depth analysis of the recent TAC 2008 update summarization results, we showed that previous automatic metrics such as ROUGE-2 and BE cannot reliably predict strong performing systems. We introduced two new terms called Correlation Recall and Correlation Precision and discussed how they cast more light on the coverage and the correctness of the respective metric. Our newly introduced metric uses only the context of named entities and definite descriptions in a summary. Linking the contexts of named entities and definite descriptions turned out to be a useful tool for predicting the quality of a summary. We hypothesize that these tuples of n- grams capture important semantic and discourse-level links between entities described by the text. Hence, these context links may also be useful for other applications such as information extraction or discourse parsing. Whether these semantic applications can benefit from context chains is left to future research. A. Confidence intervals Correlation Recall (sorted by Responsiveness) ContextChain Pearson s r ** ** ** ** upper range lower range AutoSumm Pearson s r ** ** ** ** ** upper range lower range ROUGE Pearson s r ** upper range lower range BE Pearson s r ** ** ** upper range lower range Correlation Precision (sorted by automatic metric) ContextChain Pearson s r ** ** ** ** upper range lower range AutoSumm Pearson s r ** ** ** ** upper range lower range ROUGE Pearson s r ** upper range lower range BE Pearson s r ** ** upper range lower range References [1] A. N. A. Louis. Automatic summary evaluation without human models. Proceedings of Text Understanding Conference (TAC), [2] R. Barzilay and M. Lapata. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1 34, [3] H. T. Dang. Update summarization task and opinion summarization pilot task. In Proceedings of the First Text Analysis Conference (TAC 2008), Gaithersburg, MD, Nov National Institute of Standards and Technology. [4] G. Giannakopoulos, V. Karkaletsis, G. Vouros, and P. Stamatopoulos. Summarization system evaluation revisited: N- gram graphs. ACM Trans. Speech Lang. Process., 5(3):1 39, [5] E. Hovy, C.-Y. Lin, and L. Zhou. Evaluating duc 2005 using basic elements. Proceedings of Document Understanding Conference (DUC). Vancouver, B.C., Canada, [6] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out, page 10, [7] C.-Y. Lin and E. Hovy. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages 45 51, Morristown, NJ, USA, Association for Computational Linguistics. [8] G. Loftus and E. Loftus. Essence of Statistics. McGraw Hill, 2nd edition edition, [9] P. Over, H. Dang, and D. Harman. DUC in context. Information Processing and Management, 43(6): , [10] F. Schilder, R. Kondadadi, J. L. Leidner, and J. G. Conrad. Thomson Reuters at TAC 2008: Aggressive Filtering with FastSum for Update and Opinion Summarization. In Proceedings of the First Text Analysis Conference (TAC), Gaithersburg, MD, NIST. [11] S. Tratz and E. Hovy. Summarization evaluation using transformed basic elements. Proceedings of Text Understanding Conference (TAC),

LQVSumm: A Corpus of Linguistic Quality Violations in Multi-Document Summarization

LQVSumm: A Corpus of Linguistic Quality Violations in Multi-Document Summarization LQVSumm: A Corpus of Linguistic Quality Violations in Multi-Document Summarization Annemarie Friedrich, Marina Valeeva and Alexis Palmer COMPUTATIONAL LINGUISTICS & PHONETICS SAARLAND UNIVERSITY, GERMANY

More information

HLTCOE at TREC 2013: Temporal Summarization

HLTCOE at TREC 2013: Temporal Summarization HLTCOE at TREC 2013: Temporal Summarization Tan Xu University of Maryland College Park Paul McNamee Johns Hopkins University HLTCOE Douglas W. Oard University of Maryland College Park Abstract Our team

More information

Query-based Opinion Summarization for Legal Blog Entries

Query-based Opinion Summarization for Legal Blog Entries Query-based Opinion Summarization for Legal Blog Entries Jack G. Conrad, Jochen L. Leidner, Frank Schilder, Ravi Kondadadi Research & Development Thomson Reuters Corporation St. Paul, MN 55123 USA {Jack.G.Conrad,

More information

A Case Study: News Classification Based on Term Frequency

A Case Study: News Classification Based on Term Frequency A Case Study: News Classification Based on Term Frequency Petr Kroha Faculty of Computer Science University of Technology 09107 Chemnitz Germany kroha@informatik.tu-chemnitz.de Ricardo Baeza-Yates Center

More information

Columbia University at DUC 2004

Columbia University at DUC 2004 Columbia University at DUC 2004 Sasha Blair-Goldensohn, David Evans, Vasileios Hatzivassiloglou, Kathleen McKeown, Ani Nenkova, Rebecca Passonneau, Barry Schiffman, Andrew Schlaikjer, Advaith Siddharthan,

More information

Learning Structural Correspondences Across Different Linguistic Domains with Synchronous Neural Language Models

Learning Structural Correspondences Across Different Linguistic Domains with Synchronous Neural Language Models Learning Structural Correspondences Across Different Linguistic Domains with Synchronous Neural Language Models Stephan Gouws and GJ van Rooyen MIH Medialab, Stellenbosch University SOUTH AFRICA {stephan,gvrooyen}@ml.sun.ac.za

More information

Writing a Basic Assessment Report. CUNY Office of Undergraduate Studies

Writing a Basic Assessment Report. CUNY Office of Undergraduate Studies Writing a Basic Assessment Report What is a Basic Assessment Report? A basic assessment report is useful when assessing selected Common Core SLOs across a set of single courses A basic assessment report

More information

Linking Task: Identifying authors and book titles in verbose queries

Linking Task: Identifying authors and book titles in verbose queries Linking Task: Identifying authors and book titles in verbose queries Anaïs Ollagnier, Sébastien Fournier, and Patrice Bellot Aix-Marseille University, CNRS, ENSAM, University of Toulon, LSIS UMR 7296,

More information

The Smart/Empire TIPSTER IR System

The Smart/Empire TIPSTER IR System The Smart/Empire TIPSTER IR System Chris Buckley, Janet Walz Sabir Research, Gaithersburg, MD chrisb,walz@sabir.com Claire Cardie, Scott Mardis, Mandar Mitra, David Pierce, Kiri Wagstaff Department of

More information

Variations of the Similarity Function of TextRank for Automated Summarization

Variations of the Similarity Function of TextRank for Automated Summarization Variations of the Similarity Function of TextRank for Automated Summarization Federico Barrios 1, Federico López 1, Luis Argerich 1, Rosita Wachenchauzer 12 1 Facultad de Ingeniería, Universidad de Buenos

More information

CSC200: Lecture 4. Allan Borodin

CSC200: Lecture 4. Allan Borodin CSC200: Lecture 4 Allan Borodin 1 / 22 Announcements My apologies for the tutorial room mixup on Wednesday. The room SS 1088 is only reserved for Fridays and I forgot that. My office hours: Tuesdays 2-4

More information

Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Probabilistic Latent Semantic Analysis Thomas Hofmann Presentation by Ioannis Pavlopoulos & Andreas Damianou for the course of Data Mining & Exploration 1 Outline Latent Semantic Analysis o Need o Overview

More information

Word Segmentation of Off-line Handwritten Documents

Word Segmentation of Off-line Handwritten Documents Word Segmentation of Off-line Handwritten Documents Chen Huang and Sargur N. Srihari {chuang5, srihari}@cedar.buffalo.edu Center of Excellence for Document Analysis and Recognition (CEDAR), Department

More information

SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF)

SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF) SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF) Hans Christian 1 ; Mikhael Pramodana Agus 2 ; Derwin Suhartono 3 1,2,3 Computer Science Department,

More information

Longest Common Subsequence: A Method for Automatic Evaluation of Handwritten Essays

Longest Common Subsequence: A Method for Automatic Evaluation of Handwritten Essays IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661,p-ISSN: 2278-8727, Volume 17, Issue 6, Ver. IV (Nov Dec. 2015), PP 01-07 www.iosrjournals.org Longest Common Subsequence: A Method for

More information

M55205-Mastering Microsoft Project 2016

M55205-Mastering Microsoft Project 2016 M55205-Mastering Microsoft Project 2016 Course Number: M55205 Category: Desktop Applications Duration: 3 days Certification: Exam 70-343 Overview This three-day, instructor-led course is intended for individuals

More information

Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses

Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses Thomas F.C. Woodhall Masters Candidate in Civil Engineering Queen s University at Kingston,

More information

Scoring Guide for Candidates For retake candidates who began the Certification process in and earlier.

Scoring Guide for Candidates For retake candidates who began the Certification process in and earlier. Adolescence and Young Adulthood SOCIAL STUDIES HISTORY For retake candidates who began the Certification process in 2013-14 and earlier. Part 1 provides you with the tools to understand and interpret your

More information

Extending Place Value with Whole Numbers to 1,000,000

Extending Place Value with Whole Numbers to 1,000,000 Grade 4 Mathematics, Quarter 1, Unit 1.1 Extending Place Value with Whole Numbers to 1,000,000 Overview Number of Instructional Days: 10 (1 day = 45 minutes) Content to Be Learned Recognize that a digit

More information

The Effect of Discourse Markers on the Speaking Production of EFL Students. Iman Moradimanesh

The Effect of Discourse Markers on the Speaking Production of EFL Students. Iman Moradimanesh The Effect of Discourse Markers on the Speaking Production of EFL Students Iman Moradimanesh Abstract The research aimed at investigating the relationship between discourse markers (DMs) and a special

More information

Evidence for Reliability, Validity and Learning Effectiveness

Evidence for Reliability, Validity and Learning Effectiveness PEARSON EDUCATION Evidence for Reliability, Validity and Learning Effectiveness Introduction Pearson Knowledge Technologies has conducted a large number and wide variety of reliability and validity studies

More information

arxiv: v1 [cs.cl] 2 Apr 2017

arxiv: v1 [cs.cl] 2 Apr 2017 Word-Alignment-Based Segment-Level Machine Translation Evaluation using Word Embeddings Junki Matsuo and Mamoru Komachi Graduate School of System Design, Tokyo Metropolitan University, Japan matsuo-junki@ed.tmu.ac.jp,

More information

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS

AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS AGS THE GREAT REVIEW GAME FOR PRE-ALGEBRA (CD) CORRELATED TO CALIFORNIA CONTENT STANDARDS 1 CALIFORNIA CONTENT STANDARDS: Chapter 1 ALGEBRA AND WHOLE NUMBERS Algebra and Functions 1.4 Students use algebraic

More information

Vocabulary Agreement Among Model Summaries And Source Documents 1

Vocabulary Agreement Among Model Summaries And Source Documents 1 Vocabulary Agreement Among Model Summaries And Source Documents 1 Terry COPECK, Stan SZPAKOWICZ School of Information Technology and Engineering University of Ottawa 800 King Edward Avenue, P.O. Box 450

More information

CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2

CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2 1 CROSS-LANGUAGE INFORMATION RETRIEVAL USING PARAFAC2 Peter A. Chew, Brett W. Bader, Ahmed Abdelali Proceedings of the 13 th SIGKDD, 2007 Tiago Luís Outline 2 Cross-Language IR (CLIR) Latent Semantic Analysis

More information

Using dialogue context to improve parsing performance in dialogue systems

Using dialogue context to improve parsing performance in dialogue systems Using dialogue context to improve parsing performance in dialogue systems Ivan Meza-Ruiz and Oliver Lemon School of Informatics, Edinburgh University 2 Buccleuch Place, Edinburgh I.V.Meza-Ruiz@sms.ed.ac.uk,

More information

Cross Language Information Retrieval

Cross Language Information Retrieval Cross Language Information Retrieval RAFFAELLA BERNARDI UNIVERSITÀ DEGLI STUDI DI TRENTO P.ZZA VENEZIA, ROOM: 2.05, E-MAIL: BERNARDI@DISI.UNITN.IT Contents 1 Acknowledgment.............................................

More information

THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS

THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS THE PENNSYLVANIA STATE UNIVERSITY SCHREYER HONORS COLLEGE DEPARTMENT OF MATHEMATICS ASSESSING THE EFFECTIVENESS OF MULTIPLE CHOICE MATH TESTS ELIZABETH ANNE SOMERS Spring 2011 A thesis submitted in partial

More information

A Note on Structuring Employability Skills for Accounting Students

A Note on Structuring Employability Skills for Accounting Students A Note on Structuring Employability Skills for Accounting Students Jon Warwick and Anna Howard School of Business, London South Bank University Correspondence Address Jon Warwick, School of Business, London

More information

Math-U-See Correlation with the Common Core State Standards for Mathematical Content for Third Grade

Math-U-See Correlation with the Common Core State Standards for Mathematical Content for Third Grade Math-U-See Correlation with the Common Core State Standards for Mathematical Content for Third Grade The third grade standards primarily address multiplication and division, which are covered in Math-U-See

More information

Learning Methods for Fuzzy Systems

Learning Methods for Fuzzy Systems Learning Methods for Fuzzy Systems Rudolf Kruse and Andreas Nürnberger Department of Computer Science, University of Magdeburg Universitätsplatz, D-396 Magdeburg, Germany Phone : +49.39.67.876, Fax : +49.39.67.8

More information

South Carolina English Language Arts

South Carolina English Language Arts South Carolina English Language Arts A S O F J U N E 2 0, 2 0 1 0, T H I S S TAT E H A D A D O P T E D T H E CO M M O N CO R E S TAT E S TA N DA R D S. DOCUMENTS REVIEWED South Carolina Academic Content

More information

The Internet as a Normative Corpus: Grammar Checking with a Search Engine

The Internet as a Normative Corpus: Grammar Checking with a Search Engine The Internet as a Normative Corpus: Grammar Checking with a Search Engine Jonas Sjöbergh KTH Nada SE-100 44 Stockholm, Sweden jsh@nada.kth.se Abstract In this paper some methods using the Internet as a

More information

*Net Perceptions, Inc West 78th Street Suite 300 Minneapolis, MN

*Net Perceptions, Inc West 78th Street Suite 300 Minneapolis, MN From: AAAI Technical Report WS-98-08. Compilation copyright 1998, AAAI (www.aaai.org). All rights reserved. Recommender Systems: A GroupLens Perspective Joseph A. Konstan *t, John Riedl *t, AI Borchers,

More information

Short Text Understanding Through Lexical-Semantic Analysis

Short Text Understanding Through Lexical-Semantic Analysis Short Text Understanding Through Lexical-Semantic Analysis Wen Hua #1, Zhongyuan Wang 2, Haixun Wang 3, Kai Zheng #4, Xiaofang Zhou #5 School of Information, Renmin University of China, Beijing, China

More information

Efficient Online Summarization of Microblogging Streams

Efficient Online Summarization of Microblogging Streams Efficient Online Summarization of Microblogging Streams Andrei Olariu Faculty of Mathematics and Computer Science University of Bucharest andrei@olariu.org Abstract The large amounts of data generated

More information

On the Combined Behavior of Autonomous Resource Management Agents

On the Combined Behavior of Autonomous Resource Management Agents On the Combined Behavior of Autonomous Resource Management Agents Siri Fagernes 1 and Alva L. Couch 2 1 Faculty of Engineering Oslo University College Oslo, Norway siri.fagernes@iu.hio.no 2 Computer Science

More information

WE GAVE A LAWYER BASIC MATH SKILLS, AND YOU WON T BELIEVE WHAT HAPPENED NEXT

WE GAVE A LAWYER BASIC MATH SKILLS, AND YOU WON T BELIEVE WHAT HAPPENED NEXT WE GAVE A LAWYER BASIC MATH SKILLS, AND YOU WON T BELIEVE WHAT HAPPENED NEXT PRACTICAL APPLICATIONS OF RANDOM SAMPLING IN ediscovery By Matthew Verga, J.D. INTRODUCTION Anyone who spends ample time working

More information

An NFR Pattern Approach to Dealing with Non-Functional Requirements

An NFR Pattern Approach to Dealing with Non-Functional Requirements An NFR Pattern Approach to Dealing with Non-Functional Requirements Presenter: Sam Supakkul Outline Motivation The Approach NFR Patterns Pattern Organization Pattern Reuse Tool Support Case Study Conclusion

More information

VOL. 3, NO. 5, May 2012 ISSN Journal of Emerging Trends in Computing and Information Sciences CIS Journal. All rights reserved.

VOL. 3, NO. 5, May 2012 ISSN Journal of Emerging Trends in Computing and Information Sciences CIS Journal. All rights reserved. Exploratory Study on Factors that Impact / Influence Success and failure of Students in the Foundation Computer Studies Course at the National University of Samoa 1 2 Elisapeta Mauai, Edna Temese 1 Computing

More information

Reducing Features to Improve Bug Prediction

Reducing Features to Improve Bug Prediction Reducing Features to Improve Bug Prediction Shivkumar Shivaji, E. James Whitehead, Jr., Ram Akella University of California Santa Cruz {shiv,ejw,ram}@soe.ucsc.edu Sunghun Kim Hong Kong University of Science

More information

Beyond the Blend: Optimizing the Use of your Learning Technologies. Bryan Chapman, Chapman Alliance

Beyond the Blend: Optimizing the Use of your Learning Technologies. Bryan Chapman, Chapman Alliance 901 Beyond the Blend: Optimizing the Use of your Learning Technologies Bryan Chapman, Chapman Alliance Power Blend Beyond the Blend: Optimizing the Use of Your Learning Infrastructure Facilitator: Bryan

More information

Learning Methods in Multilingual Speech Recognition

Learning Methods in Multilingual Speech Recognition Learning Methods in Multilingual Speech Recognition Hui Lin Department of Electrical Engineering University of Washington Seattle, WA 98125 linhui@u.washington.edu Li Deng, Jasha Droppo, Dong Yu, and Alex

More information

Levels of processing: Qualitative differences or task-demand differences?

Levels of processing: Qualitative differences or task-demand differences? Memory & Cognition 1983,11 (3),316-323 Levels of processing: Qualitative differences or task-demand differences? SHANNON DAWN MOESER Memorial University ofnewfoundland, St. John's, NewfoundlandAlB3X8,

More information

Major Milestones, Team Activities, and Individual Deliverables

Major Milestones, Team Activities, and Individual Deliverables Major Milestones, Team Activities, and Individual Deliverables Milestone #1: Team Semester Proposal Your team should write a proposal that describes project objectives, existing relevant technology, engineering

More information

AQUA: An Ontology-Driven Question Answering System

AQUA: An Ontology-Driven Question Answering System AQUA: An Ontology-Driven Question Answering System Maria Vargas-Vera, Enrico Motta and John Domingue Knowledge Media Institute (KMI) The Open University, Walton Hall, Milton Keynes, MK7 6AA, United Kingdom.

More information

PELLISSIPPI STATE TECHNICAL COMMUNITY COLLEGE MASTER SYLLABUS APPLIED MECHANICS MET 2025

PELLISSIPPI STATE TECHNICAL COMMUNITY COLLEGE MASTER SYLLABUS APPLIED MECHANICS MET 2025 PELLISSIPPI STATE TECHNICAL COMMUNITY COLLEGE MASTER SYLLABUS APPLIED MECHANICS MET 2025 Class Hours: 3.0 Credit Hours: 4.0 Laboratory Hours: 3.0 Revised: Fall 06 Catalog Course Description: A study of

More information

Lecturing for Deeper Learning Effective, Efficient, Research-based Strategies

Lecturing for Deeper Learning Effective, Efficient, Research-based Strategies Lecturing for Deeper Learning Effective, Efficient, Research-based Strategies An Invited Session at the 4 th Annual Celebration of Teaching Excellence at Cornell 1:30-3:00 PM on Monday 13 January 2014

More information

How to Judge the Quality of an Objective Classroom Test

How to Judge the Quality of an Objective Classroom Test How to Judge the Quality of an Objective Classroom Test Technical Bulletin #6 Evaluation and Examination Service The University of Iowa (319) 335-0356 HOW TO JUDGE THE QUALITY OF AN OBJECTIVE CLASSROOM

More information

Connect Mcgraw Hill Managerial Accounting Promo Code

Connect Mcgraw Hill Managerial Accounting Promo Code Mcgraw Hill Promo Code Free PDF ebook Download: Mcgraw Hill Promo Code Download or Read Online ebook connect mcgraw hill managerial accounting promo code in PDF Format From The Best User Guide Database

More information

Leveraging Sentiment to Compute Word Similarity

Leveraging Sentiment to Compute Word Similarity Leveraging Sentiment to Compute Word Similarity Balamurali A.R., Subhabrata Mukherjee, Akshat Malu and Pushpak Bhattacharyya Dept. of Computer Science and Engineering, IIT Bombay 6th International Global

More information

Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling

Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling Notebook for PAN at CLEF 2013 Andrés Alfonso Caurcel Díaz 1 and José María Gómez Hidalgo 2 1 Universidad

More information

Statewide Framework Document for:

Statewide Framework Document for: Statewide Framework Document for: 270301 Standards may be added to this document prior to submission, but may not be removed from the framework to meet state credit equivalency requirements. Performance

More information

Mcgraw Hill Financial Accounting Connect Promo Code

Mcgraw Hill Financial Accounting Connect Promo Code Mcgraw Hill Promo Code Free PDF ebook Download: Mcgraw Hill Promo Code Download or Read Online ebook mcgraw hill financial accounting connect promo code in PDF Format From The Best User Guide Database

More information

Classifying combinations: Do students distinguish between different types of combination problems?

Classifying combinations: Do students distinguish between different types of combination problems? Classifying combinations: Do students distinguish between different types of combination problems? Elise Lockwood Oregon State University Nicholas H. Wasserman Teachers College, Columbia University William

More information

Data Integration through Clustering and Finding Statistical Relations - Validation of Approach

Data Integration through Clustering and Finding Statistical Relations - Validation of Approach Data Integration through Clustering and Finding Statistical Relations - Validation of Approach Marek Jaszuk, Teresa Mroczek, and Barbara Fryc University of Information Technology and Management, ul. Sucharskiego

More information

Textbook Evalyation:

Textbook Evalyation: STUDIES IN LITERATURE AND LANGUAGE Vol. 1, No. 8, 2010, pp. 54-60 www.cscanada.net ISSN 1923-1555 [Print] ISSN 1923-1563 [Online] www.cscanada.org Textbook Evalyation: EFL Teachers Perspectives on New

More information

Visual CP Representation of Knowledge

Visual CP Representation of Knowledge Visual CP Representation of Knowledge Heather D. Pfeiffer and Roger T. Hartley Department of Computer Science New Mexico State University Las Cruces, NM 88003-8001, USA email: hdp@cs.nmsu.edu and rth@cs.nmsu.edu

More information

How to read a Paper ISMLL. Dr. Josif Grabocka, Carlotta Schatten

How to read a Paper ISMLL. Dr. Josif Grabocka, Carlotta Schatten How to read a Paper ISMLL Dr. Josif Grabocka, Carlotta Schatten Hildesheim, April 2017 1 / 30 Outline How to read a paper Finding additional material Hildesheim, April 2017 2 / 30 How to read a paper How

More information

PIRLS. International Achievement in the Processes of Reading Comprehension Results from PIRLS 2001 in 35 Countries

PIRLS. International Achievement in the Processes of Reading Comprehension Results from PIRLS 2001 in 35 Countries Ina V.S. Mullis Michael O. Martin Eugenio J. Gonzalez PIRLS International Achievement in the Processes of Reading Comprehension Results from PIRLS 2001 in 35 Countries International Study Center International

More information

On document relevance and lexical cohesion between query terms

On document relevance and lexical cohesion between query terms Information Processing and Management 42 (2006) 1230 1247 www.elsevier.com/locate/infoproman On document relevance and lexical cohesion between query terms Olga Vechtomova a, *, Murat Karamuftuoglu b,

More information

Instructor: Mario D. Garrett, Ph.D. Phone: Office: Hepner Hall (HH) 100

Instructor: Mario D. Garrett, Ph.D.   Phone: Office: Hepner Hall (HH) 100 San Diego State University School of Social Work 610 COMPUTER APPLICATIONS FOR SOCIAL WORK PRACTICE Statistical Package for the Social Sciences Office: Hepner Hall (HH) 100 Instructor: Mario D. Garrett,

More information

Short vs. Extended Answer Questions in Computer Science Exams

Short vs. Extended Answer Questions in Computer Science Exams Short vs. Extended Answer Questions in Computer Science Exams Alejandro Salinger Opportunities and New Directions April 26 th, 2012 ajsalinger@uwaterloo.ca Computer Science Written Exams Many choices of

More information

GROUP COMPOSITION IN THE NAVIGATION SIMULATOR A PILOT STUDY Magnus Boström (Kalmar Maritime Academy, Sweden)

GROUP COMPOSITION IN THE NAVIGATION SIMULATOR A PILOT STUDY Magnus Boström (Kalmar Maritime Academy, Sweden) GROUP COMPOSITION IN THE NAVIGATION SIMULATOR A PILOT STUDY Magnus Boström (Kalmar Maritime Academy, Sweden) magnus.bostrom@lnu.se ABSTRACT: At Kalmar Maritime Academy (KMA) the first-year students at

More information

The Importance of Social Network Structure in the Open Source Software Developer Community

The Importance of Social Network Structure in the Open Source Software Developer Community The Importance of Social Network Structure in the Open Source Software Developer Community Matthew Van Antwerp Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556

More information

VIEW: An Assessment of Problem Solving Style

VIEW: An Assessment of Problem Solving Style 1 VIEW: An Assessment of Problem Solving Style Edwin C. Selby, Donald J. Treffinger, Scott G. Isaksen, and Kenneth Lauer This document is a working paper, the purposes of which are to describe the three

More information

Measuring the relative compositionality of verb-noun (V-N) collocations by integrating features

Measuring the relative compositionality of verb-noun (V-N) collocations by integrating features Measuring the relative compositionality of verb-noun (V-N) collocations by integrating features Sriram Venkatapathy Language Technologies Research Centre, International Institute of Information Technology

More information

Language Acquisition Chart

Language Acquisition Chart Language Acquisition Chart This chart was designed to help teachers better understand the process of second language acquisition. Please use this chart as a resource for learning more about the way people

More information

A Comparison of Two Text Representations for Sentiment Analysis

A Comparison of Two Text Representations for Sentiment Analysis 010 International Conference on Computer Application and System Modeling (ICCASM 010) A Comparison of Two Text Representations for Sentiment Analysis Jianxiong Wang School of Computer Science & Educational

More information

On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC

On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC On Human Computer Interaction, HCI Dr. Saif al Zahir Electrical and Computer Engineering Department UBC Human Computer Interaction HCI HCI is the study of people, computer technology, and the ways these

More information

Hawai i Pacific University Sees Stellar Response Rates for Course Evaluations

Hawai i Pacific University Sees Stellar Response Rates for Course Evaluations Improvement at heart. CASE STUDY Hawai i Pacific University Sees Stellar Response Rates for Course Evaluations From my perspective, the company has been incredible. Without Blue, we wouldn t be able to

More information

Methods for the Qualitative Evaluation of Lexical Association Measures

Methods for the Qualitative Evaluation of Lexical Association Measures Methods for the Qualitative Evaluation of Lexical Association Measures Stefan Evert IMS, University of Stuttgart Azenbergstr. 12 D-70174 Stuttgart, Germany evert@ims.uni-stuttgart.de Brigitte Krenn Austrian

More information

Australian Journal of Basic and Applied Sciences

Australian Journal of Basic and Applied Sciences AENSI Journals Australian Journal of Basic and Applied Sciences ISSN:1991-8178 Journal home page: www.ajbasweb.com Feature Selection Technique Using Principal Component Analysis For Improving Fuzzy C-Mean

More information

Algebra 1, Quarter 3, Unit 3.1. Line of Best Fit. Overview

Algebra 1, Quarter 3, Unit 3.1. Line of Best Fit. Overview Algebra 1, Quarter 3, Unit 3.1 Line of Best Fit Overview Number of instructional days 6 (1 day assessment) (1 day = 45 minutes) Content to be learned Analyze scatter plots and construct the line of best

More information

Meta Comments for Summarizing Meeting Speech

Meta Comments for Summarizing Meeting Speech Meta Comments for Summarizing Meeting Speech Gabriel Murray 1 and Steve Renals 2 1 University of British Columbia, Vancouver, Canada gabrielm@cs.ubc.ca 2 University of Edinburgh, Edinburgh, Scotland s.renals@ed.ac.uk

More information

An Evaluation of the Interactive-Activation Model Using Masked Partial-Word Priming. Jason R. Perry. University of Western Ontario. Stephen J.

An Evaluation of the Interactive-Activation Model Using Masked Partial-Word Priming. Jason R. Perry. University of Western Ontario. Stephen J. An Evaluation of the Interactive-Activation Model Using Masked Partial-Word Priming Jason R. Perry University of Western Ontario Stephen J. Lupker University of Western Ontario Colin J. Davis Royal Holloway

More information

M-Learning. Hauptseminar E-Learning Sommersemester Michael Kellerer LFE Medieninformatik

M-Learning. Hauptseminar E-Learning Sommersemester Michael Kellerer LFE Medieninformatik M-Learning Hauptseminar E-Learning Sommersemester 2008 Michael Kellerer LFE Medieninformatik 22.07.2008 LMU Munich Media Informatics Hauptseminar SS 2008 Michael Kellerer Slide 1 / 19 Agenda Definitions

More information

Control and Boundedness

Control and Boundedness Control and Boundedness Having eliminated rules, we would expect constructions to follow from the lexical categories (of heads and specifiers of syntactic constructions) alone. Combinatory syntax simply

More information

What is PDE? Research Report. Paul Nichols

What is PDE? Research Report. Paul Nichols What is PDE? Research Report Paul Nichols December 2013 WHAT IS PDE? 1 About Pearson Everything we do at Pearson grows out of a clear mission: to help people make progress in their lives through personalized

More information

Rule Learning With Negation: Issues Regarding Effectiveness

Rule Learning With Negation: Issues Regarding Effectiveness Rule Learning With Negation: Issues Regarding Effectiveness S. Chua, F. Coenen, G. Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX Liverpool, United

More information

OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE

OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE Mark R. Shinn, Ph.D. Michelle M. Shinn, Ph.D. Formative Evaluation to Inform Teaching Summative Assessment: Culmination measure. Mastery

More information

Task Tolerance of MT Output in Integrated Text Processes

Task Tolerance of MT Output in Integrated Text Processes Task Tolerance of MT Output in Integrated Text Processes John S. White, Jennifer B. Doyon, and Susan W. Talbott Litton PRC 1500 PRC Drive McLean, VA 22102, USA {white_john, doyon jennifer, talbott_susan}@prc.com

More information

MODULE 4 Data Collection and Hypothesis Development. Trainer Outline

MODULE 4 Data Collection and Hypothesis Development. Trainer Outline MODULE 4 Data Collection and Hypothesis Development Trainer Outline The following trainer guide includes estimated times for each section of the module, an overview of the information to be presented,

More information

WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING AND TEACHING OF PROBLEM SOLVING

WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING AND TEACHING OF PROBLEM SOLVING From Proceedings of Physics Teacher Education Beyond 2000 International Conference, Barcelona, Spain, August 27 to September 1, 2000 WHY SOLVE PROBLEMS? INTERVIEWING COLLEGE FACULTY ABOUT THE LEARNING

More information

Rote rehearsal and spacing effects in the free recall of pure and mixed lists. By: Peter P.J.L. Verkoeijen and Peter F. Delaney

Rote rehearsal and spacing effects in the free recall of pure and mixed lists. By: Peter P.J.L. Verkoeijen and Peter F. Delaney Rote rehearsal and spacing effects in the free recall of pure and mixed lists By: Peter P.J.L. Verkoeijen and Peter F. Delaney Verkoeijen, P. P. J. L, & Delaney, P. F. (2008). Rote rehearsal and spacing

More information

Target Language Preposition Selection an Experiment with Transformation-Based Learning and Aligned Bilingual Data

Target Language Preposition Selection an Experiment with Transformation-Based Learning and Aligned Bilingual Data Target Language Preposition Selection an Experiment with Transformation-Based Learning and Aligned Bilingual Data Ebba Gustavii Department of Linguistics and Philology, Uppsala University, Sweden ebbag@stp.ling.uu.se

More information

Honors Mathematics. Introduction and Definition of Honors Mathematics

Honors Mathematics. Introduction and Definition of Honors Mathematics Honors Mathematics Introduction and Definition of Honors Mathematics Honors Mathematics courses are intended to be more challenging than standard courses and provide multiple opportunities for students

More information

Developing an Assessment Plan to Learn About Student Learning

Developing an Assessment Plan to Learn About Student Learning Developing an Assessment Plan to Learn About Student Learning By Peggy L. Maki, Senior Scholar, Assessing for Learning American Association for Higher Education (pre-publication version of article that

More information

PREPARING FOR THE SITE VISIT IN YOUR FUTURE

PREPARING FOR THE SITE VISIT IN YOUR FUTURE PREPARING FOR THE SITE VISIT IN YOUR FUTURE ARC-PA Suzanne York SuzanneYork@arc-pa.org 2016 PAEA Education Forum Minneapolis, MN Saturday, October 15, 2016 TODAY S SESSION WILL INCLUDE: Recommendations

More information

Audit Of Teaching Assignments. An Integrated Analysis of Teacher Educational Background and Courses Taught October 2007

Audit Of Teaching Assignments. An Integrated Analysis of Teacher Educational Background and Courses Taught October 2007 Audit Of Teaching Assignments October 2007 Audit Of Teaching Assignments Audit of Teaching Assignments Crown copyright, Province of Nova Scotia, 2007 The contents of this publication may be reproduced

More information

Practical Research. Planning and Design. Paul D. Leedy. Jeanne Ellis Ormrod. Upper Saddle River, New Jersey Columbus, Ohio

Practical Research. Planning and Design. Paul D. Leedy. Jeanne Ellis Ormrod. Upper Saddle River, New Jersey Columbus, Ohio SUB Gfittingen 213 789 981 2001 B 865 Practical Research Planning and Design Paul D. Leedy The American University, Emeritus Jeanne Ellis Ormrod University of New Hampshire Upper Saddle River, New Jersey

More information

A heuristic framework for pivot-based bilingual dictionary induction

A heuristic framework for pivot-based bilingual dictionary induction 2013 International Conference on Culture and Computing A heuristic framework for pivot-based bilingual dictionary induction Mairidan Wushouer, Toru Ishida, Donghui Lin Department of Social Informatics,

More information

Assignment 1: Predicting Amazon Review Ratings

Assignment 1: Predicting Amazon Review Ratings Assignment 1: Predicting Amazon Review Ratings 1 Dataset Analysis Richard Park r2park@acsmail.ucsd.edu February 23, 2015 The dataset selected for this assignment comes from the set of Amazon reviews for

More information

MULTILINGUAL INFORMATION ACCESS IN DIGITAL LIBRARY

MULTILINGUAL INFORMATION ACCESS IN DIGITAL LIBRARY MULTILINGUAL INFORMATION ACCESS IN DIGITAL LIBRARY Chen, Hsin-Hsi Department of Computer Science and Information Engineering National Taiwan University Taipei, Taiwan E-mail: hh_chen@csie.ntu.edu.tw Abstract

More information

College Pricing. Ben Johnson. April 30, Abstract. Colleges in the United States price discriminate based on student characteristics

College Pricing. Ben Johnson. April 30, Abstract. Colleges in the United States price discriminate based on student characteristics College Pricing Ben Johnson April 30, 2012 Abstract Colleges in the United States price discriminate based on student characteristics such as ability and income. This paper develops a model of college

More information

CHAPTER 4: REIMBURSEMENT STRATEGIES 24

CHAPTER 4: REIMBURSEMENT STRATEGIES 24 CHAPTER 4: REIMBURSEMENT STRATEGIES 24 INTRODUCTION Once state level policymakers have decided to implement and pay for CSR, one issue they face is simply how to calculate the reimbursements to districts

More information

Organizational Knowledge Distribution: An Experimental Evaluation

Organizational Knowledge Distribution: An Experimental Evaluation Association for Information Systems AIS Electronic Library (AISeL) AMCIS 24 Proceedings Americas Conference on Information Systems (AMCIS) 12-31-24 : An Experimental Evaluation Surendra Sarnikar University

More information

Summarizing Text Documents: Carnegie Mellon University 4616 Henry Street

Summarizing Text Documents:   Carnegie Mellon University 4616 Henry Street Summarizing Text Documents: Sentence Selection and Evaluation Metrics Jade Goldstein y Mark Kantrowitz Vibhu Mittal Jaime Carbonell y jade@cs.cmu.edu mkant@jprc.com mittal@jprc.com jgc@cs.cmu.edu y Language

More information

Corpus Linguistics (L615)

Corpus Linguistics (L615) (L615) Basics of Markus Dickinson Department of, Indiana University Spring 2013 1 / 23 : the extent to which a sample includes the full range of variability in a population distinguishes corpora from archives

More information

STA 225: Introductory Statistics (CT)

STA 225: Introductory Statistics (CT) Marshall University College of Science Mathematics Department STA 225: Introductory Statistics (CT) Course catalog description A critical thinking course in applied statistical reasoning covering basic

More information