Density Ratio Estimation in Machine Learning

Transcription

1 Density Ratio Estimation in Machine Learning Machine learning is an interdisciplinary field of science and engineering that studies mathematical theories and practical applications of systems that learn. This book introduces theories, methods, and applications of density ratio estimation, which is a newly emerging paradigm in the machine learning community. Various machine learning problems such as non-stationarity adaptation, outlier detection, dimensionality reduction, independent component analysis, clustering, classification, and conditional density estimation can be systematically solved via the estimation of probability density ratios. The authors offer a comprehensive introduction of various density ratio estimators including methods via density estimation, moment matching, probabilistic classification, density fitting, and density ratio fitting as well as describing how these can be applied to machine learning. The book also provides mathematical theories for density ratio estimation including parametric and non-parametric convergence analysis and numerical stability analysis to complete the first and definitive treatment of the entire framework of density ratio estimation in machine learning. Dr. Masashi Sugiyama is an Associate Professor in the Department of Computer Science at the Tokyo Institute of Technology. Dr. Taiji Suzuki is an Assistant Professor in the Department of Mathematical Informatics at the University of Tokyo, Japan. Dr. Takafumi Kanamori is an Associate Professor in the Department of Computer Science and Mathematical Informatics at Nagoya University, Japan.

2

3 Density Ratio Estimation in Machine Learning MASASHI SUGIYAMA Tokyo Institute of Technology TAIJI SUZUKI The University of Tokyo TAKAFUMI KANAMORI Nagoya University

4 cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Mexico City Cambridge University Press 32 Avenue of the Americas, New York, NY , USA Information on this title:/ Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori 2012 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2012 Printed in the United States of America A catalog record for this publication is available from the British Library. Library of Congress Cataloging in Publication data is available ISBN Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate.

5 Contents Foreword Preface page ix xi Part I Density-Ratio Approach to Machine Learning 1 Introduction Machine Learning Density-Ratio Approach to Machine Learning Algorithms of Density-Ratio Estimation Theoretical Aspects of Density-Ratio Estimation Organization of this Book at a Glance 18 Part II Methods of Density-Ratio Estimation 2 Density Estimation Basic Framework Parametric Approach Non-Parametric Approach Numerical Examples Remarks 37 3 Moment Matching Basic Framework Finite-Order Approach Infinite-Order Approach: KMM Numerical Examples Remarks 45 4 Probabilistic Classification Basic Framework Logistic Regression Least-Squares Probabilistic Classifier 50 v

6 vi Contents 4.4 Support Vector Machine Model Selection by Cross-Validation Numerical Examples Remarks 54 5 Density Fitting Basic Framework Implementations of KLIEP Model Selection by Cross-Validation Numerical Examples Remarks 65 6 Density-Ratio Fitting Basic Framework Implementation of LSIF Model Selection by Cross-Validation Numerical Examples Remarks 74 7 Unified Framework Basic Framework Existing Methods as Density-Ratio Fitting Interpretation of Density-Ratio Fitting Power Divergence for Robust Density-Ratio Estimation Remarks 87 8 Direct Density-Ratio Estimation with Dimensionality Reduction Discriminant Analysis Approach Divergence Maximization Approach Numerical Examples Remarks 115 Part III Applications of Density Ratios in Machine Learning 9 Importance Sampling Covariate Shift Adaptation Multi-Task Learning Distribution Comparison Inlier-Based Outlier Detection Two-Sample Test Mutual Information Estimation Density-Ratio Methods of Mutual Information Estimation Sufficient Dimension Reduction Independent Component Analysis 183

7 Contents vii 12 Conditional Probability Estimation Conditional Density Estimation Probabilistic Classification 203 Part IV Theoretical Analysis of Density-Ratio Estimation 13 Parametric Convergence Analysis Density-Ratio Fitting under Kullback Leibler Divergence Density-Ratio Fitting under Squared Distance Optimality of Logistic Regression Accuracy Comparison Remarks Non-Parametric Convergence Analysis Mathematical Preliminaries Non-Parametric Convergence Analysis of KLIEP Convergence Analysis of KuLSIF Remarks Parametric Two-Sample Test Introduction Estimation of Density Ratios Estimation of ASC Divergence Optimal Estimator of ASC Divergence Two-Sample Test Based on ASC Divergence Estimation Numerical Studies Remarks Non-Parametric Numerical Stability Analysis Preliminaries Relation between KuLSIF and KMM Condition Number Analysis Optimality of KuLSIF Numerical Examples Remarks 297 Part V Conclusions 17 Conclusions and Future Directions 303 List of Symbols and Abbreviations 307 References 309 Index 327

8

9 Foreword Estimating probability distributions is widely viewed as a central question in machine learning. The whole enterprise of probabilistic modeling using probabilistic graphical models is generally addressed by learning marginal and conditional probability distributions. Classification and regression starting with Fisher s fundamental contributions are similarly viewed as problems of estimating conditional densities. The present book introduces an exciting alternative perspective namely, that virtually all problems in machine learning can be formulated and solved as problems of estimating density ratios the ratios of two probability densities. This book provides a comprehensive review of the elegant line of research undertaken by the authors and their collaborators over the last decade. It reviews existing work on density-ratio estimation and derives a variety of algorithms for directly estimating density ratios. It then shows how these novel algorithms can address not only standard machine learning problems such as classification, regression, and feature selection but also a variety of other important problems such as learning under a covariate shift, multi-task learning, outlier detection, sufficient dimensionality reduction, and independent component analysis. At each point this book carefully defines the problems at hand, reviews existing work, derives novel methods, and reports on numerical experiments that validate the effectiveness and superiority of the new methods. A particularly impressive aspect of the work is that implementations of most of the methods are available for download fromthe authors web pages. The last part of the book is devoted to mathematical analyses of the methods. This includes not only an analysis for the case where the assumptions underlying the algorithms hold, but also situations in which the models are misspecified. Careful study of these results will not only provide fundamental insights into the problems and algorithms but will also provide the reader with an introduction to many valuable analytic tools. ix

10 x Foreword In summary, this is a definitive treatment of the topic of density-ratio estimation. It reflects the authors careful thinking and sustained research efforts. Researchers and students alike will find it an important source of ideas and techniques. There is no doubt that this book will change the way people think about machine learning and stimulate many new directions for research. Thomas G. Dietterich School of Electrical Engineering Oregon State University, Corvallis, OR, USA

11 Preface Machine learning is aimed at developing systems that learn. The mathematical foundation of machine learning and its real-world applications have been extensively explored in the last decades. Various tasks of machine learning, such as regression and classification, typically can be solved by estimating probability distributions behind data. However, estimating probability distributions is one of the most difficult problems in statistical data analysis, and thus solving machine learning tasks without going through distribution estimation is a key challenge in modern machine learning. So far, various algorithms have been developed that do not involve distribution estimation but solve target machine learning tasks directly. The support vector machine is a successful example that follows this line it does not estimate datagenerating distributions but directly obtains the class-decision boundary that is sufficient for classification. However, developing such an excellent algorithmfor each of the machine learning tasks could be highly costly and difficult. To overcome these limitations of current machine learning research, we introduce and develop a novel paradigmcalled density-ratio estimation instead of probability distributions, the ratio of probability densities is estimated for statistical data processing. The density-ratio approach covers various machine learning tasks, for example, non-stationarity adaptation, multi-task learning, outlier detection, two-sample tests, feature selection, dimensionality reduction, independent component analysis, causal inference, conditional density estimation, and probabilitic classification. Thus, density-ratio estimation is a versatile tool for machine learning. This book is aimed at introducing the mathematical foundation, practical algorithms, and applications of density-ratio estimation. Most of the contents of this book are based on the journal and conference papers we have published in the last couple of years. We acknowledge our collaborators for their fruitful discussions: Hirotaka Hachiya, Shohei Hido, Yasuyuki Ihara, Hisashi Kashima, Motoaki Kawanabe, Manabu Kimura, Masakazu Matsugu, Shin-ichi Nakajima, Klaus-Robert Müller, Jun Sese, Jaak Simm, Ichiro Takeuchi, Masafumi xi

12 xii Preface Picture taken in Nagano, Japan, in the summer of From left to right, Taiji Suzuki, Masashi Sugiyama, and Takafumi Kanamori. Takimoto, Yuta Tsuboi, Kazuya Ueki, Paul von Bünau, Gordon Wichern, and Makoto Yamada. Finally, we thank the Ministry of Education, Culture, Sports, Science and Technology; the Alexander von Humboldt Foundation; the Okawa Foundation; Microsoft Institute for Japanese Academic Research Collaboration Collaborative Research Project; IBM Faculty Award; Mathematisches Forschungsinstitut Oberwolfach Research-in-Pairs Program; the Asian Office of Aerospace Research and Development; Support Center for Advanced Telecommunications Technology Research Foundation; and the Japan Science and Technology Agency for their financial support. Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori