Multimodal Interactive Pattern Recognition and Applications

Transcription

1 Multimodal Interactive Pattern Recognition and Applications

2 Alejandro Héctor Toselli Enrique Vidal Francisco Casacuberta Multimodal Interactive Pattern Recognition and Applications

3 Dr. Alejandro Héctor Toselli Instituto Tecnológico de Informática Universidad Politécnica de Valencia Camino de Vera, s/n Valencia Spain Prof. Francisco Casacuberta Instituto Tecnológico de Informática Universidad Politécnica de Valencia Camino de Vera, s/n Valencia Spain Dr. Enrique Vidal Instituto Tecnológico de Informática Universidad Politécnica de Valencia Camino de Vera, s/n Valencia Spain ISBN e-isbn DOI / Springer London Dordrecht Heidelberg New York British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: Springer-Verlag London Limited 2011 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licenses issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Cover design: VTeX UAB, Lithuania Printed on acid-free paper Springer is part of Springer Science+Business Media (

4 Foreword Traditionally, the aim of pattern recognition is to automatically solve complex recognition problems. However, it has been realized that in many real world applications a correct recognition rate is needed that is higher than the one reachable with completely automatic systems. Therefore, some sort of post-processing is applied where humans correct the errors committed by machine. It turns out, however, that very often this post-processing phase is the bottleneck of a recognition system, causing most of its operational costs. The current book possesses two unique features that distinguish it from other books on Pattern Recognition. First, it proposes a radically different approach to correcting the errors committed by a system. This approach is characterized by human and machine being tied up in a much closer loop than usually. That is, the human gets involved not only after the machine has completed producing its recognition result, in order to correct errors, but during the recognition process. In this way, many errors can be avoided beforehand and correction costs can be reduced. The second unique feature of the book is that it proposes multimodal interaction between man and machine in order to correct and prevent recognition errors. Such multimodal interactions possibly include input via handwriting, speech, or gestures, in addition to the conventional input modalities of keyboard and mouse. The material of the book is presented on the basis of well founded mathematical principles, mostly Bayes theory. It includes various fundamental results that are highly original and relevant for the emerging field of interactive and multimodal pattern recognition. In addition, the book discusses in detail a number of concrete applications where interactive multimodal systems have the potential of being superior over traditional systems that consists of a recognition phase, conducted autonomously by machine, followed by a human post-processing step. Examples of such applications include unconstrained handwriting recognition, speech recognition, machine translation, text prediction, image retrieval, and parsing. To summarize, this book provides a very fresh and novel look at the whole discipline of pattern recognition. It is the first book, to my knowledge, that addresses the emerging field of interactive and multimodal systems in a unified and integrated way. This book may in fact become a standard reference for this emerging and v

5 vi Foreword fascinating new area. I highly recommend it to graduate students, academic and industrial researchers, lecturers, and practitioners working in the field of pattern recognition. Bern, Switzerland Horst Bunke

6 Preface Our interest in human computer interaction started with our participation in the TT2 project ( Trans Type-2, funded by the European Union (EU) and coordinated by Atos Origin, which dealt with the development of statistical-based technologies for computer assisted translation. Several years earlier, we had coordinated one of the first EU-funded projects on spoken machine translation (EuTrans, and, by the time TT2 started, we had already been working for years in machine translation (MT) in general. So we knew very well which was one of the major bottlenecks for the adoption of the MT technology available at that time by professional translation agencies: Many professional translators preferred to type by themselves all the text from scratch, rather than trying to take advantage of the (few) correct words of a MT-produced text, while fixing the (many) translation errors and sloppy sentences. Clearly, by post-editing the error-prone text produced by a MT system, these professionals felt they were not in command of the translation process; instead, they saw themselves just as dumb assistants of a foolish system which was producing flaky results that they had to figure out how to amend (the state of affairs about post-editing has improved over the years but the feeling of lack of control persists). In TT2 we learnt quite a few facts about the central role of human feedback in the development of assistive technologies and how this feedback can lead to great human/machine performance improvements if it is adequately taken into account in the mathematical formulation under which systems are developed. We also understood very well that, in these technologies, the traditional, accuracy-based performance criteria is not sufficiently adequate and performance has to be mainly assessed in terms of estimated human machine interaction effort. In one word, assistive technology has to be developed in such a way that the human user feels in command of the system, rather than the other way around, and human-interaction effort reduction must be the fundamental driving force behind system design. In TT2 we also started to realize that multimodal processing is somehow implicitly present in all interactive systems and that this can be advantageously exploited to improve overall system performance and usability. vii

7 viii Preface After the success of TT2, our research group (PRHLT started to look at how these ideas could be applied in many other Pattern Recognition (PR) fields, where assistive technologies are in increasing demand. As a result, we soon found ourselves coordinating a large and ambitious Spanish research program, called Multimodal Interaction in Pattern Recognition and Computer Vision (MIPRCV, This program, which involves more that 100 highly qualified Ph.D. researchers from ten research institutions, aims at developing core assistive technologies for interactive application fields as diverse as language and music processing, medical image recognition, biometrics and surveillance, advanced driving assistance systems and robotics, to name but a few. To a large extent, this book is the result of works carried out by the PRHLT research group within the MIPRCV consortium. Therefore it owes credit to many MIPRCV researchers that have directly or indirectly contributed with ideas, discussions and technical collaborations in general, as well as to all the members of PRHLT who, in one manner or another, have made it possible. These works are presented in this book in a unified way, under the PR framework of Statistical Decision Theory. First, fundamental concepts and general PR approaches for Multimodal Interaction modelling and search (or inference) are presented. Then, systems developed on the base of these concepts and approaches are described for several application fields. These include interactive transcription of handwritten and spoken documents, computer assisted language translation, interactive text generation and parsing, and relevance-based image retrieval. Finally, several prototypes developed for these applications are overviewed in the last chapter. Most of these prototypes consist in live demonstrators which can be publicly accessed through the Internet. So, readers of this book can easily try them by themselves in order to get a first-hand idea of the interesting possibilities of placing Pattern Recognition technologies within the Multimodal Interaction framework. Chapter 1 provides an introduction to Interactive Pattern Recognition, examining the challenges and research opportunities entailed by placing PR within the humaninteraction framework. Moreover, it provides an introduction to general approaches available to solve the underlying interactive search problems on the basis of existing methods to solve the corresponding non-interactive counterparts and, an overview of modern machine learning approaches which can be useful in the interactive framework. Chapter 2 establishes the common basics and framework on which are grounded the computer assisted transcription approaches described in the three subsequent Chaps.: 3, 4 and 5. On the one hand, Chaps. 3 and 5 are devoted to handwritten documents transcription providing different approaches, which cover different aspects as multimodality, user interaction ways and ergonomics, active learning, etc. On the other hand, Chap. 4 focuses directly on transcription of speech signals employing a similar approach described in Chap. 3. Likewise, Chap. 6 addresses the general topic of Interactive Machine Translation, providing an adequate human machine-interactive framework to produce highquality translation between any pair of languages. It will be shown how this also allows one to take advantage of some available multimodal interfaces to increase the

8 Preface ix productivity. Multimodal interfaces and adaptive learning in Interactive Machine Translation will be covered in Chaps. 7 and 8, respectively. With significant differences in relation to previous chapters, Chaps introduce other three Interactive Pattern Recognition topics: Interactive Parsing, Interactive Text Generation and Interactive Image Retrieval. The second one, for example, is characterized by not using input signal, whereas the first and third by not following the left-to-right protocol in the analysis of their corresponding inputs. Finally, Chap. 12 presents several full working prototypes and demonstrators of multimodal interactive pattern recognition applications. As previously commented, all of these systems serve as validating examples for the approaches that have been proposed and described throughout this book. Among other interesting things, they are designed to enable a true human computer interaction on selected tasks. Valencia, Spain E. Vidal A.H. Toselli F. Casacuberta

9 Contents 1 General Framework Introduction Classical Pattern Recognition Paradigm Decision Theory and Pattern Recognition Interactive Pattern Recognition and Multimodal Interaction Using the Human Feedback Directly Explicitly Taking Interaction History into Account Interaction with Deterministic Feedback Interactive Pattern Recognition and Decision Theory Multimodal Interaction Feedback Decoding and Adaptive Learning Interaction Protocols and Assessment General Types of Interaction Protocols Left-to-Right Interactive Predictive Processing ActiveInteraction Interaction with Weaker Feedback Interaction Without Input Data AssessingIPRSystems UserEffortEstimation IPR Search and Confidence Estimation Word Graphs Confidence Estimation Machine Learning Paradigms for IPR OnlineLearning ActiveLearning Semi-Supervised Learning ReinforcementLearning References Computer Assisted Transcription: General Framework Introduction CommonStatisticalFrameworkforHTRandASR xi

10 xii Contents 2.3 CommonStatisticalFrameworkforCATTIandCATS Adapting the Language Model Search and Decoding Methods Viterbi-BasedImplementation Word-GraphBasedImplementation AssessmentMeasures References Computer Assisted Transcription of Text Images Computer Assisted Transcription of Text Images: CATTI CATTI Search Problem Word-Graph-Based Search Approach WordGraphError-CorrectingParsing Increasing Interaction Ergonomics in CATTI: PA-CATTI Language Model and Search Multimodal Computer Assisted Transcription of Text Images: MM-CATTI Language Model and Search for MM-CATTI Non-interactiveHTRSystems MainOff-LineHTRSystemOverview On-Line HTR Subsystem Overview Tasks, Experiments and Results HTRCorpora Results Conclusions References Computer Assisted Transcription of Speech Signals ComputerAssistedTranscriptionofAudioStreams Foundations of CATS Introduction to Automatic Speech Recognition Speech Acquisition Pre-process and Feature Extraction Statistical Speech Recognition Search in CATS Word-Graph-Based CATS ErrorCorrectingPrefixParsing A General Model for Probabilistic Prefix Parsing Experimental Results Corpora ErrorMeasures Experiments Results Multimodality in CATS Experimental Results Corpora...115

11 Contents xiii Experiments Conclusions References Active Interaction and Learning in Handwritten Text Transcription Introduction Confidence Measures Adaptation from Partially Supervised Transcriptions ActiveInteractionandActiveLearning Balancing Error and Supervision Effort Experiments User Interaction Model Sequential Transcription Tasks Adaptation from Partially Supervised Transcriptions ActiveInteractionandLearning Balancing User Effort and Recognition Error Conclusions References Interactive Machine Translation Introduction Statistical Machine Translation Interactive Machine Translation Interactive Machine Translation with Confidence Estimation Search in Interactive Machine Translation Word-Graph Generation Error-CorrectingParsing Search for n-bestcompletions Tasks, Experiments and Results Pre-andPost-processing Tasks EvaluationMeasures Results Results Using Confidence Information Conclusions References Multi-Modality for Interactive Machine Translation Introduction Making Use of Weaker Feedback Non-explicitPositioningPointerActions Interaction-ExplicitPointerActions Correcting Errors with Speech Recognition Unconstrained Speech Decoding (DEC) Prefix-Conditioned Speech Decoding (DEC-PREF) Prefix-Conditioned Speech Decoding (IMT-PREF) PrefixSelection(IMT-SEL)...160

12 xiv Contents 7.4 Correcting Errors with Handwritten Text Recognition Tasks, Experiments and Results Results when Incorporating Weaker Feedback Results for Speech as Input Feedback Results for Handwritten Text as Input Feedback Conclusions References Incremental and Adaptive Learning for Interactive Machine Translation Introduction On-LineLearning Concept of On-Line Learning BasicIMTSystem OnlineIMTSystem RelatedTopics Active Learning on IMT via Confidence Measures Bayesian Adaptation Results Conclusions References Interactive Parsing Introduction InteractiveParsingFramework Confidence Measures in IP IPinLeft-to-RightDepth-FirstOrder EfficientCalculationoftheNextBestTree IP Experimentation User Simulation Subsystem EvaluationMetrics Experimental Results Conclusions References Interactive Text Generation Introduction Interactive Text Generation and Interactive Pattern Recognition Interactive Text Generation at the Word Level N-Gram Language Modeling Searching for a Suffix Optimal Greedy Prediction of Suffixes Dealing with Sentence Length Word-Level Experiments PredictingatCharacterLevel Character-Level Experiments...205

13 Contents xv 10.4 Conclusions References Interactive Image Retrieval Introduction Relevance Feedback for Image Retrieval Probabilistic Interaction Model Greedy Approximation Relevance Feedback Algorithm ASimplifiedVersionofGARF Experiments Image Feature Extraction Baseline Methods Discussion Multimodal Relevance Feedback Fusion by Refining EarlyFusion LateFusion Proposed Approach: Dynamic Linear Fusion Experiments Discussion References Prototypes and Demonstrators Introduction Passive, Left-to-Right Protocol Passive, Desultory Protocol Active Protocol PrototypeEvaluation MM-IHT: Multimodal Interactive Handwritten Transcription PrototypeDescription Technology Evaluation IST: Interactive Speech Transcription PrototypeDescription Technology Evaluation IMT: Interactive Machine Translation PrototypeDescription Technology Evaluation ITG: Interactive Text Generation PrototypeDescription Technology Evaluation MM-IP: Multimodal Interactive Parsing PrototypeDescription...251

14 xvi Contents Technology Evaluation GIDOC: GIMP-Based Interactive Document Transcription PrototypeDescription Technology Evaluation RISE: Relevant Image Search Engine PrototypeDescription Technology Evaluation Conclusions References Glossary Index...271