Model Selection and Multimodel Inference

Transcription

1 Kenneth P. Burnham David R. Anderson Model Selection and Multimodel Inference A Practical Information-Theoretic Approach Second Edition With 31 Illustrations Springer

2 Contents Preface About the Authors Glossary vii yod xxiii 1 Introduction Objectives of the Book Background Material Inference from Data, Given a Model Likelihood and Least Squares Theory The Critical Issue: "What Is the Best Model to Use?" Science Inputs: Formulation of the Set of Candidate Models Models Versus Full Reality An Ideal Approximating Model Model Fundamentals and Notation Truth or Full Reality f Approximating Models g i (x(e ) The Kullback Leibler Best Model gi (x100) Estimated Models gi(xiö) Generating Models Global Model 26

3 xiv Contents Overview of Stochastic Models in the Biological Sciences Inference and the Principle of Parsimony Avoid Overfitting to Achieve a Good Model Fit The Principle of Parsimony Model Selection Methods Data Dredging, Overanalysis of Data, and Spurious Effects Overanalysis of Data Some Trends Model Selection Bias Model Selection Uncertainty Summary 47 2 Information and Likelihood Theory: A Basis for Model Selection and Inference Kullback Leibler Information or Distance Between Two Models Examples of Kullback Leibler Distance Truth, f, Drops Out as a Constant Akaike's Information Criterion: Takeuchi's Information Criterion: Second-Order Information Criterion: Modification of Information Criterion for Overdispersed Count Data AIC Differences, A Useful Analogy Likelihood of a Model, G(gi ldata) Akaike Weights, wi Basic Formula An Extension Evidence Ratios Important Analysis Details AIC Cannot Be Used to Compare Models of Different Data Sets Order Not Important in Computing AIC Values Transformations of the Response Variable Regression Models with Differing Error Structures Do Not Mix Null Hypothesis Testing with Information-Theoretic Criteria Null Hypothesis Testing Is Still Important in Strict Experiments Information-Theoretic Criteria Are Not a "Test" Exploratory Data Analysis 84

4 Contents xv 2.12 Some History and Further Insights Entropy A Heuristic Interpretation More an Interpreting Information- Theoretic Criteria Nonnested Models Further Insights Bootstrap Methods and Model Selection Frequencies Tli Introduction The Bootstrap in Model Selection: The Basic Idea Return to Flather's Models Summary 96 3 Basic Use of the Information-Theoretic Approach Introduction Example 1: Cement Hardening Data Set of Candidate Models Some Results and Comparisons A Summary Example 2: Time Distribution of an Insecticide Added to a Simulated Ecosystem Set of Candidate Models Some Results Example 3: Nestling Starlings Experimental Scenario Monte Carlo Data Set of Candidate Models Data Analysis Results Further Insights into the First Fourteen Nested Models Hypothesis Testing and Information-Theoretic Approaches Have Different Selection Frequencies Further Insights Following Final Model Selection Why Not Always Use the Global Model for Inference? Example 4: Sage Grouse Survival Introduction Set of Candidate Models Model Selection Hypothesis Tests for Year-Dependent Survival Probabilities 131

5 xvi Contents Hypothesis Testing Versus AIC in Model Selection A Class of Intermediate Models Example 5: Resource Utilization of Anolis Lizards Set of Candidate Models Comments on Analytic Method Some Tentative Results Example 6: Sakamoto et al.'s (1986) Simulated Data Example 7: Models of Fish Growth Summary Formal Inference From More Than One Model: Multimodel Inference (MMI) Introduction to Multimodel Inference Model Averaging Prediction Averaging Across Model Parameters Model Selection Uncertainty Concepts of Parameter Estimation and Model Selection Uncertainty Including Model Selection Uncertainty in Estimator Sampling Variance Unconditional Confidence Intervals Estimating the Relative Importance of Variables Confidence Set for the K-L Best Model Introduction Ai, Model Selection Probabilities, and the Bootstrap Model Redundancy Recommendations Cement Data Pine Wood Data The Durban Storm Data Models Considered Consideration of Model Fit Confidence Intervals on Predicted Storm Probability Comparisons of Estimator Precision Flour Beetle Mortality: A Logistic Regression Example Publication of Research Results Summary Monte Carlo Insights and Extended Examples Introduction Survival Models 207

6 Contents xvii A Chain Binomial Survival Model An Example An Extended Survival Model Model Selection if Sample Size Is Huge, or Truth Known A Further Chain Binomial Model Examples and Ideas Illustrated with Linear Regression All-Subsets Selection: A GPA Example A Monte Carlo Extension of the GPA Example An Improved Set of GPA Prediction Models More Monte Carlo Results Linear Regression and Variable Selection Discussion Estimation of Density from Line Transect Sampling Density Estimation Background Line Transect Sampling of Kangaroos at Wallaby Creek Analysis of Wallaby Creek Data Bootstrap Analysis Confidence Interval an D Bootstrap Samples: 1,000 Versus 10, Bootstrap Versus Akaike Weights: A Lesson an QAIC, Summary Advanced Issues and Deeper Insights Introduction An Example with 13 Predictor Variables and 8,191 Models Body Fat Data The Global Model Classical Stepwise Selection Model Selection Uncertainty for AIC, and BIC An A Priori Approach Bootstrap Evaluation of Model Uncertainty Monte Carlo Simulations Summary Messages Overview of Model Selection Criteria Criteria That Are Estimates of K-L Information Criteria That Are Consistent for K Contrasts Consistent Selection in Practice: Quasi-true Models Contrasting AIC and BIC A Heuristic Derivation of BIC 293

7 xviii Contents A K-L-Based Conceptual Comparison of AIC and BIC Performance Comparison Exact Bayesian Model Selection Formulas Akaike Weights as Bayesian Posterior Model Probabilities Goodness-of-Fit and Overdispersion Revisited Overdispersion "e and Goodness-of-Fit: A General Strategy Overdispersion Modeling: More Than One i Model Goodness-of-Fit After Selection AIC and Random Coefficient Models Basic Concepts and Marginal Likelihood Approach A Shrinkage Approach to AIC and Random Effects On Extensions Selection When Probability Distributions Differ by Model Keep All the Parts A Normal Versus Log-Normal Example Comparing Across Several Distributions: An Example Lessons from the Literature and Other Matters Use AIC,, Not AIC, with Small Sample Sizes Use AIC e, Not AIC, When K Is Large When Is AIC, Suitable: A Gamma Distribution Example Inference from a Less Than Best Model Are Parameters Real? Sample Size Is Often Not a Simple Issue Judgment Has a Role Tidbits About AIC Irrelevance of Between-Sample Variation of AIC The G-Statistic and K-L Information AIC Versus Hypothesis Testing: Results Can Be Very Different A Subtle Model Selection Bias Issue The Dimensional Unit of AIC AIC and Finite Mixture Models Unconditional Variance A Baseline for w +(i) Summary 347

8 Contents xix 7 Statistical Theory and Numerical Results Useful Preliminaries A General Derivation of AIC General K-L-Based Model Selection: TIC Analytical Computation of TIC Bootstrap Estimation of TIC AICc : A Second-Order Improvement Derivation of AICc Lack of Uniqueness of AIC, Derivation of AIC for the Exponential Family of Distributions Evaluation of tr(j(20)[/( 0)]-1) and Its Estimator Comparison of AIC Versus TIC in a Very Simple Setting Evaluation Under Logistic Regression Evaluation Under Multinomially Distributed Count Data Evaluation Under Poisson-Distributed Data Evaluation for Fixed-Effects Normality-Based Linear Models Additional Results and Considerations Selection Simulation for Nested Models Simulation of the Distribution of Ap Does AIC Overfit? Can Selection Be Improved Based an All the A i? Linear Regression, AIC, and Mean Square Error AIC, and Models for Multivariate Data There Is No True TIC c Kullback-Leibler Information Relationship to the Fisher Information Matrix Entropy and Jaynes Maxent Principle Akaike Weights wi Versus Selection Probabilities Kullback-Leibler Information Is Always > Summary Summary The Scientific Question and the Collection of Data Actual Thinking and A Priori Modeling The Basis for Objective Model Selection The Principle of Parsimony Information Criteria as Estimates of Expected Relative Kullback-Leibler Information Ranking Alternative Models 446

9 xx Contents 8.7 Scaling Alternative Models MMI: Inference Based on Model Averaging MMI: Model Selection Uncertainty MMI: Relative Importance of Predictor Variables More on Inferences Final Thoughts 454 References 455 Index 485