Introduction to Meta-Analysis Michael Borenstein Biostat, Inc, New Jersey, USA. Larry V. Hedges Northwestern University, Evanston, USA. Julian P.T. Higgins MRC, Cambridge, UK. Hannah R. Rothstein Baruch College, New York, USA. A John Wiley and Sons, Ltd., Publication th January :50 Wiley/ITMA Page iii ffirs
This edition first published Ó John Wiley & Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 88. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 88, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloguing-in-Publication Data Introduction to meta-analysis / Michael Borenstein... [et al.]. p. ; cm. Includes bibliographical references and index. ISBN 978-0-470-7-7 (cloth) 1. Meta-analysis. I. Borenstein, Michael. [DNLM: 1. Meta-Analysis as Topic. WA 950 I6 ]. R853.M48I58 6.72 dc A catalogue record for this book is available from the British Library. ISBN: 978-0-470-7-7 Set in.5/pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed in the UK by TJ International, Padstow, Cornwall th January :50 Wiley/ITMA Page iv ffirs
Contents List of Tables List of Figures Acknowledgements Preface Web site xiii xv xix xxi xxix PART 1: INTRODUCTION 1 HOW A META-ANALYSIS WORKS 3 Introduction 3 Individual studies 3 The summary effect 5 Heterogeneity of effect sizes 6 Summary points 7 2 WHY PERFORM A META-ANALYSIS 9 Introduction 9 The streptokinase meta-analysis Statistical significance Clinical importance of the effect Consistency of effects Summary points PART 2: EFFECT SIZE AND PRECISION 3 OVERVIEW Treatment effects and effect sizes Parameters and estimates Outline of effect size computations 4 EFFECT SIZES BASED ON MEANS Introduction Raw (unstandardized) mean difference D Standardized mean difference, d and g Response ratios Summary points th February : Wiley/ITMA Page v ftoc
vi Contents 5 EFFECT SIZES BASED ON BINARY DATA (2 2 TABLES) Introduction Risk ratio Odds ratio Risk difference Choosing an effect size index Summary points 6 EFFECT SIZES BASED ON CORRELATIONS Introduction Computing r Other approaches Summary points 7 CONVERTING AMONG EFFECT SIZES 45 Introduction 45 Converting from the log odds ratio to d 47 Converting from d to the log odds ratio 47 Converting from r to d 48 Converting from d to r 48 Summary points 49 8 FACTORS THAT AFFECT PRECISION 51 Introduction 51 Factors that affect precision 52 Sample size 52 Study design 53 Summary points 55 9 CONCLUDING REMARKS 57 PART 3: FIXED-EFFECT VERSUS RANDOM-EFFECTS MODELS OVERVIEW 61 Introduction 61 Nomenclature 62 FIXED-EFFECT MODEL 63 Introduction 63 The true effect size 63 Impact of sampling error 63 th February : Wiley/ITMA Page vi ftoc
Contents vii Performing a fixed-effect meta-analysis 65 Summary points 67 RANDOM-EFFECTS MODEL 69 Introduction 69 The true effect sizes 69 Impact of sampling error 70 Performing a random-effects meta-analysis 72 Summary points 74 FIXED-EFFECT VERSUS RANDOM-EFFECTS MODELS 77 Introduction 77 Definition of a summary effect 77 Estimating the summary effect 78 Extreme effect size in a large study or a small study 79 Confidence interval 80 The null hypothesis 83 Which model should we use? 83 Model should not be based on the test for heterogeneity 84 Concluding remarks 85 Summary points 85 WORKED EXAMPLES (PART 1) 87 Introduction 87 Worked example for continuous data (Part 1) 87 Worked example for binary data (Part 1) 92 Worked example for correlational data (Part 1) 97 Summary points 2 PART 4: HETEROGENEITY OVERVIEW 5 Introduction 5 Nomenclature 6 Worked examples 6 IDENTIFYING AND QUANTIFYING HETEROGENEITY 1 Introduction 1 Isolating the variation in true effects 1 Computing Q 1 Estimating 2 4 The I 2 statistic 7 th February : Wiley/ITMA Page vii ftoc
viii Contents Comparing the measures of heterogeneity 9 Confidence intervals for 2 1 Confidence intervals (or uncertainty intervals) for I 2 1 Summary points 1 PREDICTION INTERVALS 1 Introduction 1 Prediction intervals in primary studies 1 Prediction intervals in meta-analysis 1 Confidence intervals and prediction intervals 1 Comparing the confidence interval with the prediction interval 1 Summary points 3 WORKED EXAMPLES (PART 2) 1 Introduction 1 Worked example for continuous data (Part 2) 1 Worked example for binary data (Part 2) 1 Worked example for correlational data (Part 2) 1 Summary points 7 SUBGROUP ANALYSES 9 Introduction 9 Fixed-effect model within subgroups 1 Computational models 1 Random effects with separate estimates of 2 4 Random effects with pooled estimate of 2 1 The proportion of variance explained 9 Mixed-effects model 3 Obtaining an overall effect in the presence of subgroups 4 Summary points 6 META-REGRESSION 7 Introduction 7 Fixed-effect model 8 Fixed or random effects for unexplained heterogeneity 3 Random-effects model 6 Summary points 2 NOTES ON SUBGROUP ANALYSES AND META-REGRESSION 2 Introduction 2 Computational model 2 Multiple comparisons 2 Software 2 Analyses of subgroups and regression analyses are observational 2 th February : Wiley/ITMA Page viii ftoc
Contents ix Statistical power for subgroup analyses and meta-regression 0 Summary points 1 PART 5: COMPLEX DATA STRUCTURES OVERVIEW 5 INDEPENDENT SUBGROUPS WITHIN A STUDY 7 Introduction 7 Combining across subgroups 8 Comparing subgroups 2 Summary points 2 MULTIPLE OUTCOMES OR TIME-POINTS WITHIN A STUDY 2 Introduction 2 Combining across outcomes or time-points 2 Comparing outcomes or time-points within a study 3 Summary points 2 MULTIPLE COMPARISONS WITHIN A STUDY 2 Introduction 2 Combining across multiple comparisons within a study 2 Differences between treatments 2 Summary points 2 NOTES ON COMPLEX DATA STRUCTURES 2 Introduction 2 Summary effect 2 Differences in effect 4 PART 6: OTHER ISSUES OVERVIEW 9 VOTE COUNTING A NEW NAME FOR AN OLD PROBLEM 1 Introduction 1 Why vote counting is wrong 2 Vote counting is a pervasive problem 3 Summary points 5 POWER ANALYSIS FOR META-ANALYSIS 7 Introduction 7 A conceptual approach 7 In context 1 When to use power analysis 2 th February : Wiley/ITMA Page ix ftoc
x Contents Planning for precision rather than for power 3 Power analysis in primary studies 3 Power analysis for meta-analysis 7 Power analysis for a test of homogeneity 2 Summary points 5 PUBLICATION BIAS 7 Introduction 7 The problem of missing studies 8 Methods for addressing bias 0 Illustrative example 1 The model 1 Getting a sense of the data 1 Is there evidence of any bias? 3 Is the entire effect an artifact of bias? 4 How much of an impact might the bias have? 6 Summary of the findings for the illustrative example 9 Some important caveats 0 Small-study effects 1 Concluding remarks 1 Summary points 1 PART 7: ISSUES RELATED TO EFFECT SIZE OVERVIEW 5 EFFECT SIZES RATHER THAN p -VALUES 7 Introduction 7 Relationship between p-values and effect sizes 7 The distinction is important 9 The p-value is often misinterpreted 0 Narrative reviews vs. meta-analyses 1 Summary points 2 SIMPSON S PARADOX 3 Introduction 3 Circumcision and risk of HIV infection 3 An example of the paradox 5 Summary points 8 GENERALITY OF THE BASIC INVERSE-VARIANCE METHOD 1 Introduction 1 Other effect sizes 2 Other methods for estimating effect sizes 5 Individual participant data meta-analyses 6 th February : Wiley/ITMA Page x ftoc
Contents xi Bayesian approaches 8 Summary points 9 PART 8: FURTHER METHODS OVERVIEW 3 META-ANALYSIS METHODS BASED ON DIRECTION AND p -VALUES 5 Introduction 5 Vote counting 5 The sign test 5 Combining p-values 6 Summary points 3 FURTHER METHODS FOR DICHOTOMOUS DATA 3 Introduction 3 Mantel-Haenszel method 3 One-step (Peto) formula for odds ratio 3 Summary points 3 PSYCHOMETRIC META-ANALYSIS 3 Introduction 3 The attenuating effects of artifacts 3 Meta-analysis methods 4 Example of psychometric meta-analysis 6 Comparison of artifact correction with meta-regression 8 Sources of information about artifact values 9 How heterogeneity is assessed 9 Reporting in psychometric meta-analysis 0 Concluding remarks 1 Summary points 1 PART 9: META-ANALYSIS IN CONTEXT OVERVIEW 5 WHEN DOES IT MAKE SENSE TO PERFORM A META-ANALYSIS? 7 Introduction 7 Are the studies similar enough to combine? 8 Can I combine studies with different designs? 9 How many studies are enough to carry out a meta-analysis? 3 Summary points 4 REPORTING THE RESULTS OF A META-ANALYSIS 5 Introduction 5 The computational model 6 th February : Wiley/ITMA Page xi ftoc
xii Contents Forest plots 6 Sensitivity analysis 8 Summary points 9 CUMULATIVE META-ANALYSIS 1 Introduction 1 Why perform a cumulative meta-analysis? 3 Summary points 6 CRITICISMS OF META-ANALYSIS 7 Introduction 7 One number cannot summarize a research field 8 The file drawer problem invalidates meta-analysis 8 Mixing apples and oranges 9 Garbage in, garbage out 0 Important studies are ignored 1 Meta-analysis can disagree with randomized trials 1 Meta-analyses are performed poorly 4 Is a narrative review better? 5 Concluding remarks 6 Summary points 6 PART : RESOURCES AND SOFTWARE 44 SOFTWARE 1 Introduction 1 The software 2 Three examples of meta-analysis software 3 Comprehensive Meta-Analysis (CMA) 2.0 5 RevMan 5.0 8 Stata macros with Stata.0 0 Summary points 3 45 BOOKS, WEB SITES AND PROFESSIONAL ORGANIZATIONS 5 Books on systematic review methods 5 Books on meta-analysis 5 Web sites 6 REFERENCES 9 INDEX 5 th February : Wiley/ITMA Page xii ftoc