Ensemble Methods. Zhi-Hua Zhou. Foundations and Algorithms. Chapman & Hall/CRC. CRC Press. Machine Learning & Pattern Recognition Series

Transcription

1 Chapman & Hall/CRC Machine Learning & Pattern Recognition Series Ensemble Methods Foundations and Algorithms Zhi-Hua Zhou CRC Press Taylor & Francis Croup Boca Raton London New York CRC Press is an imprint of the Taylor & Francis Croup, an Informa business A CHAPMAN 6c HALL BOOK

2 Preface vii Notations ix... 1 Introduction Basic Concepts Popular Learning Algorithms Linear Discriminant Analysis Decision Trees Neural Networks Naive Bayes Classifier fc-nearest Neighbor Support Vector Machines and Kernel Methods Evaluation and Comparison Ensemble Methods Applications of Ensemble Methods Further Readings 20 2 Boosting A General Boosting Procedure The AdaBoost Algorithm Illustrative Examples Theoretical Issues Initial Analysis Margin Explanation Statistical View Multiclass Extension Noise Tolerance Further Readings 44 3 Bagging Two Ensemble Paradigms The Bagging Algorithm Illustrative Examples Theoretical Issues Random Tree Ensembles Random Forest 57 xi

3 xii Spectrum of Randomization Random Tree Ensembles for Density Estimation Random Tree Ensembles for Anomaly Detection Further Readings 66 4 Combination Methods Benefits of Combination Averaging Simple Averaging Weighted Averaging Voting Majority Voting Plurality Voting Weighted Voting Soft Voting Theoretical Issues Combining by Learning Stacking Infinite Ensemble Other Combination Methods Algebraic Methods Behavior Knowledge Space Method Decision Template Method Relevant Methods Error-Correcting Output Codes Dynamic Classifier Selection Mixture of Experts Further Readings 95 5 Diversity Ensemble Diversity Error Decomposition Error-Ambiguity Decomposition Bias-Variance-Covariance Decomposition Diversity Measures Pairwise Measures Non-Pairwise Measures Summary and Visualization Limitation of Diversity Measures Information Theoretic Diversity Ill Information Theory and Ensemble Ill Interaction Information Diversity Multi-Information Diversity Estimation Method 114 ^.5 Diversity Generation 116

4 xiii 5.6 Further Readings Ensemble Pruning What Is Ensemble Pruning Many Could Be BetterThan All Categorization of Pruning Methods Ordering-Based Pruning Clustering-Based Pruning Optimization-Based Pruning Heuristic Optimization Pruning Mathematical Programming Pruning Probabilistic Pruning Further Readings Clustering Ensembles Clustering Clustering Methods Clustering Evaluation Why Clustering Ensembles Categorization of Clustering Ensemble Methods Similarity-Based Methods Graph-Based Methods Relabeling-Based Methods Transformation-Based Methods Further Readings Advanced Topics Semi-Supervised Learning Usefulness of Unlabeled Data Semi-Supervised Learning with Ensembles Active Learning Usefulness of Human Intervention Active Learning with Ensembles Cost-Sensitive Learning Learning with Unequal Costs Ensemble Methods for Cost-Sensitive. Learning 8.4 Class-Imbalance Learning Learning with Class Imbalance Performance Evaluation with Class Imbalance Ensemble Methods for Class-Imbalance Learning 8.5 Improving Comprehensibility Reduction ofensemble to Single Model Rule Extraction from Ensembles Visualization ofensembles Future Directions of Ensembles 182

5 xiv 8.7 Further Readings References Index