AN INVESTIGATION OF THE BENCHMARK. Khosrow Kaikhah. Department of Computer Science. Southwest Texas State University. San Marcos, Texas 78666
|
|
- Marshall Burns
- 6 years ago
- Views:
Transcription
1 AN INVESTIGATION OF THE BENCHMARK EVALUATION TOOL Khosrow Kaikhah Department of Computer Science Southwest Texas State University San Marcos, Texas Final Report for: Summer Research Program Rome Laboratory Sponsored by: Air Force Office of Scientific Research Bolling Air Force Base, Washington, D.C. August 1992
2 AN INVESTIGATION OF THE BENCHMARK EVALAUATION TOOL K. Kaikhah Department of Computer Science Southwest Texas State University ABSTRACT Recently, natural language processing has received tremendous support and popularity. As a consequence, the number of natural language processing systems has dramatically increased and the need for a systematic evaluation procedure of such systems seems inevitable. Until recently, there has not been a universal evaluation procedure for evaluating all types of NLP systems. Evaluations of such systems are usually conducted during the implementation phase and, in most cases, do not involve a comprehensive plan or independent evaluators. Developers of NLP systems can benefit from an unbiased evaluation procedure which measures their efforts and the power of their systems. At the same time, the consumers of NLP systems can benefit greatly from an evaluation tool which assists with the selection of the appropriate system for their needs. The Calspan Corporation has proposed and implemented the Benchmark Evaluation Tool for evaluating all natural language processing systems, regardless of type or application. The study was sponsored by the Rome Laboratory and was concluded in May of The Benchmark Evaluation Tool is designed to be domain independent. Therefore, it concentrates on the linguistic issues rather than on the application domain. This feature is unique, in that, the tool is sensitive to each individual linguistic capability and not to each individual application. It is composed of twelve independent sections which are designed to progressively test different linguistic features of NLP systems. The Benchmark Evaluation Tool also includes definitions and explanations for each section as well as a five-choice scoring strategy to measure the responses. Our objective is to investigate the effectiveness of the Benchmark Evaluation Tool by applying the tool to a natural language processing system. This particular system is composed of two major parts: a domain-independent part which has general knowledge of syntactic rules, and a domainspecific part which provides the necessary semantic and pragmatic knowledge for a specified domain. The application domain accompanying the NLP system for testing purposes is an interface to a relational database of air travel planning irifonnation.
3 AN INVESTIGATION OF THE BENCHMARK EVALAUATION TOOL K. Kaikhah 1. Introduction Although natural language processing has been on the minds of researchers from the early days of the inception of digital computers, it has never enjoyed such a tremendous popularity and support as it has received over the past two decades. As the number of natural language processing systems has increased, so has the need for a systematic evaluation procedure for testing NLP systems. Both producers and consumers of NLP systems can benefit from a well defined evaluation procedure. It can help the producers with conducting an unbiased evaluation of their systems, and can help the consumers with choosing the appropriate system for their needs. The evaluation procedure should not be defined for a particular system, but rather as a blueprint for testing the linguistics features of NLP systems. Until recently, evaluation procedures have been implemented and administrated by the developers of NLP systems. As a result, evaluations tend to be biased and follow known success patterns. These patterns may not be deliberate, but nevertheless it is the result of being so involved with the development. Therefore, a number of NLP researchers and consumers have expressed their needs and desires for an unbiased and independent evaluation procedure. One should keep in mind that a universal evaluation procedure which can be applied to all NLP systems may be too ambitious. However, a foundation for evaluating systems can be laid out to guide the producers, consumers, as well as the independent evaluators through the evaluation. The Benchmark Investigation/Identification program sponsored by the Rome Laboratory developed an evaluation tool and application procedure for evaluating natural language processing systems. The duration of the project was eighteen months; it was completed in May of It produced an evaluation procedure consisting of twelve sections. Each section is designed to test a different linguistics capability of NLP systems and provides brief explanations and definitions of the linguistic feature being tested, patterns that define the structure of the test sentence, example sentences, and criteria against which to evaluate the behavior of the NLP system. Each test sentence is then scored according to the level of system's comprehension. It can range from success (8) to Partial success (P) to No output (N). For more details, see [1].
4 Most applications of NLP systems involve interactive human-computer interfaces which include: a) Data Base Management Systems, b) Command and Control Systems, c) Decision- Aiding Systems, d) Engineering Design Systems, and e) Diagnostic Systems. The natural language processing system which is used for this investigation is equipped with an interface to a relational database. The system can respond to questions about ground transportation, fares, and flights for the cities of Atlanta, Boston, Baltimore, Denver, Dallas, Fort Worth, Pittsburgh, Philadelphia, Oakland, San Francisco, and Washington D.C. The NLP system analyzes the English sentences with three independent modules syntactic, semantic, and pragmatic in order to transform the sentences into application calls. Twenty four different switches control the behavior of the NLP system. By setting the appropriate switches, the system can be prompted to learn unknown grammatical structures and words. This process, however, requires a knowledgeable linguistic trainer, since the NLP system expects meaningful linguistic feedback during training. The parse tree as well as the semantic, and pragmatic analysis of sentences can also be examined, if so desired, by setting the appropriate switches. The goal of this investigation has been to determine the feasibility and usefulness of a universal evaluation procedure, namely the Benchmark Evaluation Tool. The Benchmark Evaluation Tool is designed to be applicable to all types of NLP systems, therefore, it can be considered to be a universal evaluation tool. We have applied the Benchmark Evaluation Tool to an NLP system and the comprehensive results are included in section 4. The following sections briefly describe the Benchmark Evaluation Tool and the NLP system, respectively. 2. The Benchmark Evaluation Tool In May of 1992, a Rome Laboratory sponsored project, The Benchmark Investigation/Identification Program was completed by Calspan Advanced Technology Center and their subcontractor, Language Systems Incorporated. The goal of the project was to develop a standard evaluation tool which is domain-independent and which can be applied to all NLP systems, regardless of their types, and without any need for modifying or porting the NLP system to a test domain. For more details, see [1]. There are several areas in which NLP systems can be evaluated. They include: a) linguistic competence, b) end user issues such as reliability and likeability, c) system development issues such as maintainability and portability, and d) intelligent behavior issues such as learning and cooperative dialogue. The Benchmark Evaluation Tool has focused on linguistic competence of NLP systems including lexical, syntactic, semantic, and discourse capabilities. It consists of twelve
5 sections, each of which tests a different feature of the NLP systems. They are: I) Basic Sentences, II) Interrogative Sentences, III) Noun Phrases, IV) Adverbials, V) Verbs and Verb Phrases, VI) Quantifiers, VII) Comparatives, VIII) Connectives, IX) Embedded Sentences, X) Reference, XI) Ellipsis, and XII) Semantics of Events. The Evaluation Tool is designed for people with non-linguistics backgrounds, therefore it provides instructions and explanatory materials for each section. These materials are provided to assist the evaluators with the creation or tailoring of test sentences and do not include a set of predefined natural language test sentences. The testing is conducted in a progressive manner from elementary sentence types to more complex sentence types. This strategy allows the evaluator to concentrate on a single linguistic feature in each test sentence. If the NLP system fails on a certain linguistic feature, the evaluator is advised not to include the feature in subsequent test sentences. The scoring is done according to the following criteria [1]: Success (S): The system successfully met the evaluation criteria stated for the particular test item. Correct (C): The system did not successfully meet the evaluation criteria, but produced acceptable/correct output. Partially Correct (P): The system did not successfully meet the evaluation criteria, and only produced partially acceptable/correct output. Failure (F): The system did not successfully meet the evaluation criteria and produced no correct output. No Output (N): The system produced no output. In short, the Benchmark Evaluation Tool is a procedure that a) produces profiles of NLP systems which are descriptive, hierarchically organized, quantitative, and objective, b) is usable across domains and applications, c) is usable across the different types of NLP systems, and d) is unbiased with respect to linguistic theories and does not require an evaluator who is a trained linguist. In fact, the Evaluation Tool is unique in two features [1]: The profiling facility Its usability and applicability across domains and applications
6 3. The NLP System The NLP system, used in our investigation, consists of two major components: a domainindependent (core) component, and a domain-specific (application) component. The domainindependent routines which include the procedural components for syntactic, semantic, and pragmatic analysis as well as large portions of the grammar and lexicon do not change during porting. However the domain-specific routines which include specialized lexicon, semantic rules, knowledge base, and application-specific routines must be re-implemented to accommodate a new application. Three distinct modules syntactic, semantic, and pragmatic analyze and process the input sentences independently. The syntactic module requires knowledge of lexicon and grammar rules; the semantic module requires the services of semantics rules; and the pragmatic module requires domain knowledge. The NLP system contains twenty four switches which control its behavior. The behavior of each module can be seen by setting the appropriate switches. In certain modes, however, the switches only control what may appear on the screen and not the processing that is going on in the background. Input to the system can be from external files or keyboard. The output can take several forms depending on the configuration of the switches. The syntactic analysis of the sentences can produce two types of output: a detailed surface structure parse tree, and an operator-argument representation called the Intermediate Syntactic Representation (ISR). ISR is the simplified version of the parse tree with a single canonical form for a number of various surface structures and a lot less detail that is not required by the semantic analyzer. The semantic and pragmatic modules use the ISR as input and produce the Integrated Discourse Representation (lor). IDRs are application-neutral representations of the meaning of the sentences in the current discourse containing situations described in the input sentences, the entities referred to, and the way the entities participate in the situations. 4. Applying the Benchmark Evaluation Tool to the NLP system The application domain which accompanied the NLP system for this investigation is an interface to a relational database of air travel planning information. Test sentences were scored according to the syntactic, semantic, and pragmatic processing and comprehension of each sentence. Generally, the sentences which failed to produce an lor (i.e. failed the analysis of the
7
8
9
10
11
12
13
14
15
16
17
18
19
20 5 Conclusions Although the idea of testing the sensitivity of individual linguistic capabilities of NLP systems rather than the sensitivity of the systems to individual applications is extremely attractive, it has nevertheless proved to be an ambitious task. Most NLP systems are designed for well-defined domains and applications. Therefore, a general purpose evaluation tool may not be suitable for all types of NLP systems. This was evident from our investigation. The NLP system, in our investigation, has an extremely narrow application domain which responds fairly well to sentences that satisfy its requirements, however the sentences that do not, fail to be analyzed completely. Each type of NLP system possesses certain attributes that are unique. Each type has strengths and weaknesses which are directly associated to the goals and objectives of the system. Therefore, the evaluation procedure should be more sensitive to the type of the NLP system being evaluated. For instance, if the NLP system is a Data Base Management System, the evaluation tool must place more emphasis on the interrogative and basic sentences rather than on quantifiers and ellipsis. Since after all, the system is not designed to respond to ellipsis or quantifiers. Some of the grammar patterns suggested by the Benchmark Evaluation Tool are not used in everyday conversation, however they are perfectly correct. For instance, 'List the flights which are more expensive than the Boston to Atlanta flight is expensive' (VII-1.1) is grammatically correct, but the second part of the sentence 'flight is expensive' is normally omitted and is implied by the first part. This may cause some confusion among some of the evaluators. In addition, scoring may also pose some confusion, since the boundaries between suggested scores are not well defined and are subjective. Hence, two independent evaluators may score a single NLP system completely different. Although not all suggested sentence patterns were applicable, nevertheless they helped with defining the boundaries of the NLP system. In many instances such as VII-3.1, VII-3.2, VII-9.1, and VII-9.2 the clash between the wide scope of the evaluation tool and the narrow application domain of the NLP system was clearly evident. In defense of the NLP system, it must be noted that no NLP system can successfully satisfy all the rigorous requirements of the Benchmark Evaluation Tool. The Benchmark Evaluation Tool proved to be extremely helpful in providing guidance and structure for evaluating the NLP system, therefore it should be used as a guide to select the appropriate testing procedures for individual types of systems, rather than as a general purpose evaluation procedure that can be applied to all NLP systems...
21 Strengths of the Benchmark Evaluation Tool: Comprehensive Contains detailed explanations Independent of NLP systems and their application domain... Defines the boundaries of NLP systems Weaknesses of the Benchmark Evaluation Tool: Time consuming Scope of the evaluation is too wide Some suggested patterns seem unusual and are not used in everyday conversation... Scoring is not well defined In conclusion, the evaluation process proved to be extremely time consuming. It is conceivable that in the near future the evaluation process could be fully automated. However, in order for an automated evaluator to be successful, the evaluation should be performed in a narrower space with well-defined boundaries. Therefore, there should be several different automated evaluators each specialized for a different type of NLP system. Each automated evaluator would have syntactic, semantic, and pragmatic knowledge of only one type of NLP system and would generate appropriate test sentences. The set of automated evaluators would form a complete collection of tools for evaluating all types of NLP systems. The Benchmark Evaluation Tool will be extremely instrumental in developing the automated evaluators. 6. References [1]. Benchmark Investigation/Identification Program Volume I; Final Report, Calspan Advanced Technology Center, P.O. Box 400, Buffalo, NY 14225, May 1992.
On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC
On Human Computer Interaction, HCI Dr. Saif al Zahir Electrical and Computer Engineering Department UBC Human Computer Interaction HCI HCI is the study of people, computer technology, and the ways these
More informationParsing of part-of-speech tagged Assamese Texts
IJCSI International Journal of Computer Science Issues, Vol. 6, No. 1, 2009 ISSN (Online): 1694-0784 ISSN (Print): 1694-0814 28 Parsing of part-of-speech tagged Assamese Texts Mirzanur Rahman 1, Sufal
More informationAQUA: An Ontology-Driven Question Answering System
AQUA: An Ontology-Driven Question Answering System Maria Vargas-Vera, Enrico Motta and John Domingue Knowledge Media Institute (KMI) The Open University, Walton Hall, Milton Keynes, MK7 6AA, United Kingdom.
More informationMaximizing Learning Through Course Alignment and Experience with Different Types of Knowledge
Innov High Educ (2009) 34:93 103 DOI 10.1007/s10755-009-9095-2 Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Phyllis Blumberg Published online: 3 February
More informationAn Introduction to the Minimalist Program
An Introduction to the Minimalist Program Luke Smith University of Arizona Summer 2016 Some findings of traditional syntax Human languages vary greatly, but digging deeper, they all have distinct commonalities:
More informationSome Principles of Automated Natural Language Information Extraction
Some Principles of Automated Natural Language Information Extraction Gregers Koch Department of Computer Science, Copenhagen University DIKU, Universitetsparken 1, DK-2100 Copenhagen, Denmark Abstract
More informationNatural Language Processing. George Konidaris
Natural Language Processing George Konidaris gdk@cs.brown.edu Fall 2017 Natural Language Processing Understanding spoken/written sentences in a natural language. Major area of research in AI. Why? Humans
More informationCS 598 Natural Language Processing
CS 598 Natural Language Processing Natural language is everywhere Natural language is everywhere Natural language is everywhere Natural language is everywhere!"#$%&'&()*+,-./012 34*5665756638/9:;< =>?@ABCDEFGHIJ5KL@
More informationInitial English Language Training for Controllers and Pilots. Mr. John Kennedy École Nationale de L Aviation Civile (ENAC) Toulouse, France.
Initial English Language Training for Controllers and Pilots Mr. John Kennedy École Nationale de L Aviation Civile (ENAC) Toulouse, France Summary All French trainee controllers and some French pilots
More informationSpecification and Evaluation of Machine Translation Toy Systems - Criteria for laboratory assignments
Specification and Evaluation of Machine Translation Toy Systems - Criteria for laboratory assignments Cristina Vertan, Walther v. Hahn University of Hamburg, Natural Language Systems Division Hamburg,
More informationAn Interactive Intelligent Language Tutor Over The Internet
An Interactive Intelligent Language Tutor Over The Internet Trude Heift Linguistics Department and Language Learning Centre Simon Fraser University, B.C. Canada V5A1S6 E-mail: heift@sfu.ca Abstract: This
More informationJacqueline C. Kowtko, Patti J. Price Speech Research Program, SRI International, Menlo Park, CA 94025
DATA COLLECTION AND ANALYSIS IN THE AIR TRAVEL PLANNING DOMAIN Jacqueline C. Kowtko, Patti J. Price Speech Research Program, SRI International, Menlo Park, CA 94025 ABSTRACT We have collected, transcribed
More informationImplementing a tool to Support KAOS-Beta Process Model Using EPF
Implementing a tool to Support KAOS-Beta Process Model Using EPF Malihe Tabatabaie Malihe.Tabatabaie@cs.york.ac.uk Department of Computer Science The University of York United Kingdom Eclipse Process Framework
More informationA GENERIC SPLIT PROCESS MODEL FOR ASSET MANAGEMENT DECISION-MAKING
A GENERIC SPLIT PROCESS MODEL FOR ASSET MANAGEMENT DECISION-MAKING Yong Sun, a * Colin Fidge b and Lin Ma a a CRC for Integrated Engineering Asset Management, School of Engineering Systems, Queensland
More informationOn-Line Data Analytics
International Journal of Computer Applications in Engineering Sciences [VOL I, ISSUE III, SEPTEMBER 2011] [ISSN: 2231-4946] On-Line Data Analytics Yugandhar Vemulapalli #, Devarapalli Raghu *, Raja Jacob
More informationSyntax Parsing 1. Grammars and parsing 2. Top-down and bottom-up parsing 3. Chart parsers 4. Bottom-up chart parsing 5. The Earley Algorithm
Syntax Parsing 1. Grammars and parsing 2. Top-down and bottom-up parsing 3. Chart parsers 4. Bottom-up chart parsing 5. The Earley Algorithm syntax: from the Greek syntaxis, meaning setting out together
More informationHandbook for Graduate Students in TESL and Applied Linguistics Programs
Handbook for Graduate Students in TESL and Applied Linguistics Programs Section A Section B Section C Section D M.A. in Teaching English as a Second Language (MA-TESL) Ph.D. in Applied Linguistics (PhD
More informationCreating Travel Advice
Creating Travel Advice Classroom at a Glance Teacher: Language: Grade: 11 School: Fran Pettigrew Spanish III Lesson Date: March 20 Class Size: 30 Schedule: McLean High School, McLean, Virginia Block schedule,
More informationProcedures for Academic Program Review. Office of Institutional Effectiveness, Academic Planning and Review
Procedures for Academic Program Review Office of Institutional Effectiveness, Academic Planning and Review Last Revision: August 2013 1 Table of Contents Background and BOG Requirements... 2 Rationale
More informationSyllabus of the Course Skills for the Tourism Industry
Syllabus of the Course Skills for the Tourism Industry IDENTIFICATION Subject Name: Skills for the Tourism Industry Code: 090 Degree: Degree in Tourism Academic year: 07 8 Subject Type: Compulsory ECTS
More informationTEKS Correlations Proclamation 2017
and Skills (TEKS): Material Correlations to the Texas Essential Knowledge and Skills (TEKS): Material Subject Course Publisher Program Title Program ISBN TEKS Coverage (%) Chapter 114. Texas Essential
More informationTraining Catalogue for ACOs Global Learning Services V1.2. amadeus.com
Training Catalogue for ACOs Global Learning Services V1.2 amadeus.com Global Learning Services Training Catalogue for ACOs V1.2 This catalogue lists the training courses offered to ACOs by Global Learning
More informationA Minimalist Approach to Code-Switching. In the field of linguistics, the topic of bilingualism is a broad one. There are many
Schmidt 1 Eric Schmidt Prof. Suzanne Flynn Linguistic Study of Bilingualism December 13, 2013 A Minimalist Approach to Code-Switching In the field of linguistics, the topic of bilingualism is a broad one.
More informationJEFFERSON COLLEGE COURSE SYLLABUS BUS 261 BUSINESS COMMUNICATIONS. 3 Credit Hours. Prepared by: Cindy Rossi January 25, 2014
JEFFERSON COLLEGE COURSE SYLLABUS BUS 261 BUSINESS COMMUNICATIONS 3 Credit Hours Prepared by: Cindy Rossi January 25, 2014 Ms. Linda Abernathy, Math, Science and Business Division Chair Ms. Shirley Davenport,
More informationAN INTRODUCTION (2 ND ED.) (LONDON, BLOOMSBURY ACADEMIC PP. VI, 282)
B. PALTRIDGE, DISCOURSE ANALYSIS: AN INTRODUCTION (2 ND ED.) (LONDON, BLOOMSBURY ACADEMIC. 2012. PP. VI, 282) Review by Glenda Shopen _ This book is a revised edition of the author s 2006 introductory
More informationDifferent Requirements Gathering Techniques and Issues. Javaria Mushtaq
835 Different Requirements Gathering Techniques and Issues Javaria Mushtaq Abstract- Project management is now becoming a very important part of our software industries. To handle projects with success
More informationApplications of memory-based natural language processing
Applications of memory-based natural language processing Antal van den Bosch and Roser Morante ILK Research Group Tilburg University Prague, June 24, 2007 Current ILK members Principal investigator: Antal
More informationInstrumentation, Control & Automation Staffing. Maintenance Benchmarking Study
Electronic Document Instrumentation, Control & Automation Staffing Prepared by ITA Technical Committee, Maintenance Subcommittee, Task Force on IC&A Staffing John Petito, Chair Richard Haugh, Vice-Chair
More informationSOFTWARE EVALUATION TOOL
SOFTWARE EVALUATION TOOL Kyle Higgins Randall Boone University of Nevada Las Vegas rboone@unlv.nevada.edu Higgins@unlv.nevada.edu N.B. This form has not been fully validated and is still in development.
More informationInleiding Taalkunde. Docent: Paola Monachesi. Blok 4, 2001/ Syntax 2. 2 Phrases and constituent structure 2. 3 A minigrammar of Italian 3
Inleiding Taalkunde Docent: Paola Monachesi Blok 4, 2001/2002 Contents 1 Syntax 2 2 Phrases and constituent structure 2 3 A minigrammar of Italian 3 4 Trees 3 5 Developing an Italian lexicon 4 6 S(emantic)-selection
More informationChapter 9 Banked gap-filling
Chapter 9 Banked gap-filling This testing technique is known as banked gap-filling, because you have to choose the appropriate word from a bank of alternatives. In a banked gap-filling task, similarly
More informationBachelor of Arts. Intercultural German Studies. Language in intercultural contexts
Bachelor of Arts Intercultural German Studies Language in intercultural contexts The programme Intercultural German Studies in Bayreuth fosters interdisciplinary thinking. It s a great choice for students
More informationPROCESS USE CASES: USE CASES IDENTIFICATION
International Conference on Enterprise Information Systems, ICEIS 2007, Volume EIS June 12-16, 2007, Funchal, Portugal. PROCESS USE CASES: USE CASES IDENTIFICATION Pedro Valente, Paulo N. M. Sampaio Distributed
More informationUSER ADAPTATION IN E-LEARNING ENVIRONMENTS
USER ADAPTATION IN E-LEARNING ENVIRONMENTS Paraskevi Tzouveli Image, Video and Multimedia Systems Laboratory School of Electrical and Computer Engineering National Technical University of Athens tpar@image.
More informationEQuIP Review Feedback
EQuIP Review Feedback Lesson/Unit Name: On the Rainy River and The Red Convertible (Module 4, Unit 1) Content Area: English language arts Grade Level: 11 Dimension I Alignment to the Depth of the CCSS
More informationMODULE 4 Data Collection and Hypothesis Development. Trainer Outline
MODULE 4 Data Collection and Hypothesis Development Trainer Outline The following trainer guide includes estimated times for each section of the module, an overview of the information to be presented,
More informationEnglish Language and Applied Linguistics. Module Descriptions 2017/18
English Language and Applied Linguistics Module Descriptions 2017/18 Level I (i.e. 2 nd Yr.) Modules Please be aware that all modules are subject to availability. If you have any questions about the modules,
More informationIntra-talker Variation: Audience Design Factors Affecting Lexical Selections
Tyler Perrachione LING 451-0 Proseminar in Sound Structure Prof. A. Bradlow 17 March 2006 Intra-talker Variation: Audience Design Factors Affecting Lexical Selections Abstract Although the acoustic and
More informationCriterion Met? Primary Supporting Y N Reading Street Comprehensive. Publisher Citations
Program 2: / Arts English Development Basic Program, K-8 Grade Level(s): K 3 SECTIO 1: PROGRAM DESCRIPTIO All instructional material submissions must meet the requirements of this program description section,
More informationOKLAHOMA 4-H SHOOTING SPORTS POLICY Revised June 2010 Revised June 2007 Original 1994
OKLAHOMA 4-H SHOOTING SPORTS POLICY Revised June 2010 Revised June 2007 Original 1994 I. SITUATION STATEMENT The Oklahoma 4-H Shooting Sports Program started in 1982. During that time it developed into
More informationObjectives. Chapter 2: The Representation of Knowledge. Expert Systems: Principles and Programming, Fourth Edition
Chapter 2: The Representation of Knowledge Expert Systems: Principles and Programming, Fourth Edition Objectives Introduce the study of logic Learn the difference between formal logic and informal logic
More informationDerivational: Inflectional: In a fit of rage the soldiers attacked them both that week, but lost the fight.
Final Exam (120 points) Click on the yellow balloons below to see the answers I. Short Answer (32pts) 1. (6) The sentence The kinder teachers made sure that the students comprehended the testable material
More informationFinal Teach For America Interim Certification Program
Teach For America Interim Certification Program Program Rubric Overview The Teach For America (TFA) Interim Certification Program Rubric was designed to provide formative and summative feedback to TFA
More informationCollege of Liberal Arts (CLA)
College of Liberal Arts (CLA) 1 College of Liberal Arts (CLA) Courses CLA 1001. The CLA First Year Experience. 1 Credit Hour. The CLA First Year Experience introduces students to the rich diversity of
More information5. UPPER INTERMEDIATE
Triolearn General Programmes adapt the standards and the Qualifications of Common European Framework of Reference (CEFR) and Cambridge ESOL. It is designed to be compatible to the local and the regional
More informationThe College Board Redesigned SAT Grade 12
A Correlation of, 2017 To the Redesigned SAT Introduction This document demonstrates how myperspectives English Language Arts meets the Reading, Writing and Language and Essay Domains of Redesigned SAT.
More informationThe Oregon Literacy Framework of September 2009 as it Applies to grades K-3
The Oregon Literacy Framework of September 2009 as it Applies to grades K-3 The State Board adopted the Oregon K-12 Literacy Framework (December 2009) as guidance for the State, districts, and schools
More informationEducating Georgia s Future gadoe.org. Richard Woods, Georgia s School Superintendent. Richard Woods, Georgia s School Superintendent. gadoe.
Serving 13 th Annual Federal Programs Conference June 18-19, 2015 Title II, Part A Workshop Sharon Brown Pam Daniels 6/18/2015 1 Topics Equitable Participation Consultation Professional Development Guidance
More informationDeveloping True/False Test Sheet Generating System with Diagnosing Basic Cognitive Ability
Developing True/False Test Sheet Generating System with Diagnosing Basic Cognitive Ability Shih-Bin Chen Dept. of Information and Computer Engineering, Chung-Yuan Christian University Chung-Li, Taiwan
More informationGrammars & Parsing, Part 1:
Grammars & Parsing, Part 1: Rules, representations, and transformations- oh my! Sentence VP The teacher Verb gave the lecture 2015-02-12 CS 562/662: Natural Language Processing Game plan for today: Review
More informationCELTA. Syllabus and Assessment Guidelines. Third Edition. University of Cambridge ESOL Examinations 1 Hills Road Cambridge CB1 2EU United Kingdom
CELTA Syllabus and Assessment Guidelines Third Edition CELTA (Certificate in Teaching English to Speakers of Other Languages) is accredited by Ofqual (the regulator of qualifications, examinations and
More informationACCOUNTING FOR LAWYERS SYLLABUS
ACCOUNTING FOR LAWYERS SYLLABUS PROF. WILLIS OFFICE: 331 PHONE: 352-273-0680 (TAX OFFICE) OFFICE HOURS: Wednesday 10:00 2:00 (for Tax Timing) plus Tuesday/Thursday from 1:00 4:00 (all classes). Email:
More informationSchool Inspection in Hesse/Germany
Hessisches Kultusministerium School Inspection in Hesse/Germany Contents 1. Introduction...2 2. School inspection as a Procedure for Quality Assurance and Quality Enhancement...2 3. The Hessian framework
More informationA Pilot Study on Pearson s Interactive Science 2011 Program
Final Report A Pilot Study on Pearson s Interactive Science 2011 Program Prepared by: Danielle DuBose, Research Associate Miriam Resendez, Senior Researcher Dr. Mariam Azin, President Submitted on August
More informationeportfolio Trials in Three Systems: Training Requirements for Campus System Administrators, Faculty, and Students
eportfolio Trials in Three Systems: Training Requirements for Campus System Administrators, Faculty, and Students Mary Bold, Ph.D., CFLE, Associate Professor, Texas Woman s University Corin Walker, M.S.,
More informationAbstractions and the Brain
Abstractions and the Brain Brian D. Josephson Department of Physics, University of Cambridge Cavendish Lab. Madingley Road Cambridge, UK. CB3 OHE bdj10@cam.ac.uk http://www.tcm.phy.cam.ac.uk/~bdj10 ABSTRACT
More informationFirst Grade Curriculum Highlights: In alignment with the Common Core Standards
First Grade Curriculum Highlights: In alignment with the Common Core Standards ENGLISH LANGUAGE ARTS Foundational Skills Print Concepts Demonstrate understanding of the organization and basic features
More informationSER CHANGES~ACCOMMODATIONS PAGES
EAST PARISH SCHOOL BOARD EXCEPTIONAL STUDENT SERVICES DEPARTMENT Excellence in Education! 12732 SILLIMAN STREET. P.O. BOX 397 CLINTON, LOUISIANA 70722 PHONE: (225) 683-8582 FAX: (225) 683-8525 www.efpsb.k12.la.us
More informationStimulating Techniques in Micro Teaching. Puan Ng Swee Teng Ketua Program Kursus Lanjutan U48 Kolej Sains Kesihatan Bersekutu, SAS, Ulu Kinta
Stimulating Techniques in Micro Teaching Puan Ng Swee Teng Ketua Program Kursus Lanjutan U48 Kolej Sains Kesihatan Bersekutu, SAS, Ulu Kinta Learning Objectives General Objectives: At the end of the 2
More informationCommon Core State Standards for English Language Arts
Reading Standards for Literature 6-12 Grade 9-10 Students: 1. Cite strong and thorough textual evidence to support analysis of what the text says explicitly as well as inferences drawn from the text. 2.
More informationCHAPTER XI DIRECT TESTIMONY OF REGINALD M. AUSTRIA ON BEHALF OF SOUTHERN CALIFORNIA GAS COMPANY AND SAN DIEGO GAS & ELECTRIC COMPANY
Application No: A.1-09-00 Exhibit No.: Witness: R. Austria Application of Southern California Gas Company (U 90 G) and San Diego Gas & Electric Company (U 90 G) to Recover Costs Recorded in the Pipeline
More informationA student diagnosing and evaluation system for laboratory-based academic exercises
A student diagnosing and evaluation system for laboratory-based academic exercises Maria Samarakou, Emmanouil Fylladitakis and Pantelis Prentakis Technological Educational Institute (T.E.I.) of Athens
More informationIntensive English Program Southwest College
Intensive English Program Southwest College ESOL 0352 Advanced Intermediate Grammar for Foreign Speakers CRN 55661-- Summer 2015 Gulfton Center Room 114 11:00 2:45 Mon. Fri. 3 hours lecture / 2 hours lab
More informationSoftware Maintenance
1 What is Software Maintenance? Software Maintenance is a very broad activity that includes error corrections, enhancements of capabilities, deletion of obsolete capabilities, and optimization. 2 Categories
More informationL1 and L2 acquisition. Holger Diessel
L1 and L2 acquisition Holger Diessel Schedule Comparing L1 and L2 acquisition The role of the native language in L2 acquisition The critical period hypothesis [student presentation] Non-linguistic factors
More informationBANGLA TO ENGLISH TEXT CONVERSION USING OPENNLP TOOLS
Daffodil International University Institutional Repository DIU Journal of Science and Technology Volume 8, Issue 1, January 2013 2013-01 BANGLA TO ENGLISH TEXT CONVERSION USING OPENNLP TOOLS Uddin, Sk.
More informationCEFR Overall Illustrative English Proficiency Scales
CEFR Overall Illustrative English Proficiency s CEFR CEFR OVERALL ORAL PRODUCTION Has a good command of idiomatic expressions and colloquialisms with awareness of connotative levels of meaning. Can convey
More informationLING 329 : MORPHOLOGY
LING 329 : MORPHOLOGY TTh 10:30 11:50 AM, Physics 121 Course Syllabus Spring 2013 Matt Pearson Office: Vollum 313 Email: pearsonm@reed.edu Phone: 7618 (off campus: 503-517-7618) Office hrs: Mon 1:30 2:30,
More informationAuthor: Justyna Kowalczys Stowarzyszenie Angielski w Medycynie (PL) Feb 2015
Author: Justyna Kowalczys Stowarzyszenie Angielski w Medycynie (PL) www.angielskiwmedycynie.org.pl Feb 2015 Developing speaking abilities is a prerequisite for HELP in order to promote effective communication
More informationHARPER ADAMS UNIVERSITY Programme Specification
HARPER ADAMS UNIVERSITY Programme Specification 1 Awarding Institution: Harper Adams University 2 Teaching Institution: Askham Bryan College 3 Course Accredited by: Not Applicable 4 Final Award and Level:
More informationKentucky s Standards for Teaching and Learning. Kentucky s Learning Goals and Academic Expectations
Kentucky s Standards for Teaching and Learning Included in this section are the: Kentucky s Learning Goals and Academic Expectations Kentucky New Teacher Standards (Note: For your reference, the KDE website
More informationAdvanced Grammar in Use
Advanced Grammar in Use A self-study reference and practice book for advanced learners of English Third Edition with answers and CD-ROM cambridge university press cambridge, new york, melbourne, madrid,
More informationEDUC-E328 Science in the Elementary Schools
1 INDIANA UNIVERSITY NORTHWEST School of Education EDUC-E328 Science in the Elementary Schools Time: Monday 9 a.m. to 3:45 Place: Instructor: Matthew Benus, Ph.D. Office: Hawthorn Hall 337 E-mail: mbenus@iun.edu
More informationBy Laurence Capron and Will Mitchell, Boston, MA: Harvard Business Review Press, 2012.
Copyright Academy of Management Learning and Education Reviews Build, Borrow, or Buy: Solving the Growth Dilemma By Laurence Capron and Will Mitchell, Boston, MA: Harvard Business Review Press, 2012. 256
More informationCourse Syllabus Advanced-Intermediate Grammar ESOL 0352
Semester with Course Reference Number (CRN) Course Syllabus Advanced-Intermediate Grammar ESOL 0352 Fall 2016 CRN: (10332) Instructor contact information (phone number and email address) Office Location
More informationAccounting 380K.6 Accounting and Control in Nonprofit Organizations (#02705) Spring 2013 Professors Michael H. Granof and Gretchen Charrier
Accounting 380K.6 Accounting and Control in Nonprofit Organizations (#02705) Spring 2013 Professors Michael H. Granof and Gretchen Charrier 1. Office: Prof Granof: CBA 4M.246; Prof Charrier: GSB 5.126D
More informationThe SREB Leadership Initiative and its
SREB LEADERSHIP INITIATIVE SREB s Leadership Curriculum Modules Engage Leaders in Solving Real School Problems Every school has leadership that results in improved student performance and leadership begins
More informationAN ERROR ANALYSIS ON THE USE OF DERIVATION AT ENGLISH EDUCATION DEPARTMENT OF UNIVERSITAS MUHAMMADIYAH YOGYAKARTA. A Skripsi
AN ERROR ANALYSIS ON THE USE OF DERIVATION AT ENGLISH EDUCATION DEPARTMENT OF UNIVERSITAS MUHAMMADIYAH YOGYAKARTA A Skripsi Submitted to the Faculty of Language Education in a Partial Fulfillment of the
More informationModule 12. Machine Learning. Version 2 CSE IIT, Kharagpur
Module 12 Machine Learning 12.1 Instructional Objective The students should understand the concept of learning systems Students should learn about different aspects of a learning system Students should
More informationMinimalism is the name of the predominant approach in generative linguistics today. It was first
Minimalism Minimalism is the name of the predominant approach in generative linguistics today. It was first introduced by Chomsky in his work The Minimalist Program (1995) and has seen several developments
More informationMYP Language A Course Outline Year 3
Course Description: The fundamental piece to learning, thinking, communicating, and reflecting is language. Language A seeks to further develop six key skill areas: listening, speaking, reading, writing,
More informationColorado State University Department of Construction Management. Assessment Results and Action Plans
Colorado State University Department of Construction Management Assessment Results and Action Plans Updated: Spring 2015 Table of Contents Table of Contents... 2 List of Tables... 3 Table of Figures...
More informationProgram in Molecular Medicine
Graduate Program in Life Sciences Program in Molecular Medicine Student and Faculty Handbook 2017-2018 UNIVERSITY OF MARYLAND GRADUATE SCHOOL UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE Graduate Program
More informationProcedia - Social and Behavioral Sciences 154 ( 2014 )
Available online at www.sciencedirect.com ScienceDirect Procedia - Social and Behavioral Sciences 154 ( 2014 ) 263 267 THE XXV ANNUAL INTERNATIONAL ACADEMIC CONFERENCE, LANGUAGE AND CULTURE, 20-22 October
More informationDeveloping a TT-MCTAG for German with an RCG-based Parser
Developing a TT-MCTAG for German with an RCG-based Parser Laura Kallmeyer, Timm Lichte, Wolfgang Maier, Yannick Parmentier, Johannes Dellert University of Tübingen, Germany CNRS-LORIA, France LREC 2008,
More informationA Context-Driven Use Case Creation Process for Specifying Automotive Driver Assistance Systems
A Context-Driven Use Case Creation Process for Specifying Automotive Driver Assistance Systems Hannes Omasreiter, Eduard Metzker DaimlerChrysler AG Research Information and Communication Postfach 23 60
More informationHigher education is becoming a major driver of economic competitiveness
Executive Summary Higher education is becoming a major driver of economic competitiveness in an increasingly knowledge-driven global economy. The imperative for countries to improve employment skills calls
More informationCommunication around Interactive Tables
Communication around Interactive Tables Figure 1. Research Framework. Izdihar Jamil Department of Computer Science University of Bristol Bristol BS8 1UB, UK Izdihar.Jamil@bris.ac.uk Abstract Despite technological,
More informationChamilo 2.0: A Second Generation Open Source E-learning and Collaboration Platform
Chamilo 2.0: A Second Generation Open Source E-learning and Collaboration Platform doi:10.3991/ijac.v3i3.1364 Jean-Marie Maes University College Ghent, Ghent, Belgium Abstract Dokeos used to be one of
More informationAn Industrial Technologist s Core Knowledge: Web-based Strategy for Defining Our Discipline
Volume 17, Number 2 - February 2001 to April 2001 An Industrial Technologist s Core Knowledge: Web-based Strategy for Defining Our Discipline By Dr. John Sinn & Mr. Darren Olson KEYWORD SEARCH Curriculum
More informationMBA 5652, Research Methods Course Syllabus. Course Description. Course Material(s) Course Learning Outcomes. Credits.
MBA 5652, Research Methods Course Syllabus Course Description Guides students in advancing their knowledge of different research principles used to embrace organizational opportunities and combat weaknesses
More informationUser education in libraries
International Journal of Library and Information Science Vol. 1(1) pp. 001-005 June, 2009 Available online http://www.academicjournals.org/ijlis 2009 Academic Journals Review User education in libraries
More informationAutomating the E-learning Personalization
Automating the E-learning Personalization Fathi Essalmi 1, Leila Jemni Ben Ayed 1, Mohamed Jemni 1, Kinshuk 2, and Sabine Graf 2 1 The Research Laboratory of Technologies of Information and Communication
More informationSecond Language Acquisition in Adults: From Research to Practice
Second Language Acquisition in Adults: From Research to Practice Donna Moss, National Center for ESL Literacy Education Lauren Ross-Feldman, Georgetown University Second language acquisition (SLA) is the
More informationWriting Research Articles
Marek J. Druzdzel with minor additions from Peter Brusilovsky University of Pittsburgh School of Information Sciences and Intelligent Systems Program marek@sis.pitt.edu http://www.pitt.edu/~druzdzel Overview
More informationDelaware Performance Appraisal System Building greater skills and knowledge for educators
Delaware Performance Appraisal System Building greater skills and knowledge for educators DPAS-II Guide for Administrators (Assistant Principals) Guide for Evaluating Assistant Principals Revised August
More informationTIMSS ADVANCED 2015 USER GUIDE FOR THE INTERNATIONAL DATABASE. Pierre Foy
TIMSS ADVANCED 2015 USER GUIDE FOR THE INTERNATIONAL DATABASE Pierre Foy TIMSS Advanced 2015 orks User Guide for the International Database Pierre Foy Contributors: Victoria A.S. Centurino, Kerry E. Cotter,
More informationAC : DEVELOPMENT OF AN INTRODUCTION TO INFRAS- TRUCTURE COURSE
AC 2011-746: DEVELOPMENT OF AN INTRODUCTION TO INFRAS- TRUCTURE COURSE Matthew W Roberts, University of Wisconsin, Platteville MATTHEW ROBERTS is an Associate Professor in the Department of Civil and Environmental
More informationBENG Simulation Modeling of Biological Systems. BENG 5613 Syllabus: Page 1 of 9. SPECIAL NOTE No. 1:
BENG 5613 Syllabus: Page 1 of 9 BENG 5613 - Simulation Modeling of Biological Systems SPECIAL NOTE No. 1: Class Syllabus BENG 5613, beginning in 2014, is being taught in the Spring in both an 8- week term
More informationSpeech Recognition at ICSI: Broadcast News and beyond
Speech Recognition at ICSI: Broadcast News and beyond Dan Ellis International Computer Science Institute, Berkeley CA Outline 1 2 3 The DARPA Broadcast News task Aspects of ICSI
More information