Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Fundamentals of Statistical Exponential Families with Applications in Statistical Decision Theory Lawrence D. Brown Cornell University
Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Shanti S. Gupta, Series Editor Volume 9 Fundamentals of Statistical Exponential Families with Applications in Statistical Decision Theory Lawrence D. Brown Cornell University Institute of Mathematical Statistics Hayward, California
Institute of Mathematical Statistics Lecture Notes-Monograph Series Series Editor, Shanti S. Gupta, Purdue University The production of the IMS Lecture Notes-Monograph Series is managed by the IMS Business Office: Nicholas P. Jewell, Treasurer, and Jose L. Gonzalez, Business Manager. Library of Congress Catalog Card Number: 87-80020 International Standard Book Number 0-940600-10-2 Copyright 1986 Institute of Mathematical Statistics All rights reserved Printed in the United States of America
To my family for their love and understanding
PREFACE I first met exponential families as a beginning graduate student. The previous summer I had written a short research report under the direction of Richard Bellman at the RAND Corporation. That report was about a dynamic programming problem concerning sequential observation of binomial variables. Jack Kiefer read that report. He conjectured that the properties of the binomial distribution used there were properties shared by all "Koopman-Darmois" distributions. (This is a name sometimes used for exponential families, in honor of the authors of two of the pioneering papers on the topic. See Koopman (1936), and Darmois (1935), and also Pitman (1936).) Jack suggested that I recast the paper into the Koopman-Darmois setting. That suggestion had two objectives. One was the hope that viewing the problem from this general perspective would lead to a clearer understanding of its structure and perhaps a simpler and better proof. The other objective was the hope of generalizing the result from the binomial to other classes of distributions, for example the Poisson and the gamma. (The resulting manuscript appeared as Brown (1965).) These two objectives of clearer understanding and of possible generalization in statistical applications are the motivation for this monograph. Many if not most of the successful mathematical formulations of statistical questions involve specific exponential families of distributions such as the normal, the exponential and gamma, the beta, the binomial and the multinomial, the geometric and the negative binomial, and the Poisson among others. It is often informative and advantageous to view these mathematical formulations
vi PREFACE from the perspective of general exponential families. These notes provide a systematic treatment of the analytic and probabilistic properties of exponential families. This treatment is constructed with a variety of statistical applications in mind. This basic theory appears in Chapters 1-3, 5, 6 and the first part of Chapter 7 (through Section 7.11). Chapter 4, the latter part of Chapter 7, and many of the examples and exercises elsewhere in the text develop selected statistical applications of the basic theory. Almost all the specific statistical applications presented here are within the area of statistical decision theory. However, as suggested above the scope of application of exponential families is much wider yet. They are, for further example, a valuable tool in asymptotic statistical theory. The presentation of the basic theory here was designed to be also suitable for applications in this area. Exercises 2.19.1, 5.15.1-5.15.4 and 7.5.1-7.5.5 provide further background for some of these applications. Efron (1975) gives an elegant example of what can be done in this area. Some earlier treatments of the general topic have proved helpful to me and have influenced my presentation, both consciously and unconsciously. The most important of these is Barndorff-Nielsen (1978). The latter half of that book treats many of the same topics as the current monograph, although they are arranged differently and presented from a different point-of-view. Lehmann (1959) contains an early definitive treatment of some fundamental results such as Theorems 1.13, 2.2, 2.7 and 2.12. Rockafellar (1970) treats in great detail the duality theory which appears in Chapters 5 and 6. I found Johansen (1979) also to be useful, particularly in the preparation of Chapter 1. The first version of this monograph was prepared during a year's leave at the Technion, Haifa, and the second was prepared during a temporary appointment at the Hebrew University, Jerusalem. I wish to express my gratitude to both those institutions and especially to my colleagues in both departments for their hospitality, interest, and encouragement. I also want to acknowledge
PREFACE vii the support from the National Science Foundation which I received throughout the preparation of this manuscript. I am grateful to all the colleagues and students who have heard me lecture on the contents or have read versions of this monograph. Nearly all have made measurable, positive contributions. Among these I want to specially thank Richard Ellis, Jiunn Hwang, Iain Johnstone, John Marden, and Yossi Rinott who have particularly influenced specific portions of the text, Jim Berger who made numerous valuable suggestions, and above all Roger Farrell who carefully read and critically and constructively commented on the entire manuscript. The draft version of the index was prepared by Fu-Hsieng Hsieh. Finally, I want to thank the editor of this series, Shanti Gupta, for his gentle but persistent encouragement which made an important contribution to the completion of this monograph.
TABLE OF CONTENTS CHAPTER 1. BASIC PROPERTIES 1 Standard Exponential Families 1 Marginal Distributions 8 Reduction to a Minimal Family 13 Random Samples 16 Convexity Property 19 Conditional Distributions 21 Exercises 26 CHAPTER 2. ANALYTIC PROPERTIES 32 Differentiability and Moments 32 Formulas for Moments 34 Analyticity 38 Completeness 42 Mutual Independence 44 Continuity Theorem 48 Total Positivity 53 Partial Order Properties 57 Exercises 60 CHAPTER 3. PARAMETRIZATIONS 70 Steep Families 70 Mean Value Parametrization 73 Mixed Parametrization 78 IX
x TABLE OF CONTENTS Differentiate Subfamilies 81 Exercises 85 CHAPTER 4. APPLICATIONS 90 Information Inequality 90 Unbiased Estimates of the Risk 99 Generalized Bayes Estimators of Canonical Parameters 106 Generalized Bayes Estimators of Expectation Parameters; Conjugate Priors 112 Exercises 124 CHAPTER 5. MAXIMUM LIKELIHOOD ESTIMATION 144 Full Families 148 Non-Full Families 152 Convex Parameter Space 153 Fundamental Equation 160 Exercises 167 CHAPTER 6. THE DUAL TO THE MAXIMUM LIKELIHOOD ESTIMATOR 174 Convex Duality 178 Minimum Entropy Parameter 184 Aggregate Exponential Families 191 Exercises 203 CHAPTER 7. TAIL PROBABILITIES 208 Fixed Parameter (Via Chebyshev's Inequality) 208 Fixed Parameter (Via Kullback-Leibler Information) 212 Fixed Reference Set 214 Complete Class Theorems for Tests (Separated Hypotheses) 220 Complete Class Theorems for Tests (Contiguous Hypotheses) 232 Exercises 239 APPENDIX TO CHAPTER 4. POINTWISE LIMITS OF BAYES PROCEDURES 254 REFERENCES 269 INDEX 280