Coding Ockham s Razor

Size: px

Start display at page:

Download "Coding Ockham s Razor"

Estella Holland
5 years ago
Views:

1 Coding Ockham s Razor

2 Lloyd Allison Coding Ockham s Razor 123

3 Lloyd Allison Faculty of Information Technology Monash University Melbourne, Victoria, Australia ISBN ISBN (ebook) Library of Congress Control Number: Springer International Publishing AG, part of Springer Nature 2018 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printed on acid-free paper This Springer imprint is published by the registered company Springer International Publishing AG part of Springer Nature. The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

4 To Sally, Bridget, Jean, Yeshi, Nyima, and Lhamo.

5 Preface The minimum message length (MML) principle was devised by Chris Wallace ( ) and David Boulton in the late 1960s [12, 93] initially to solve the unsupervised mixture modelling problem an important problem, a mathematical analysis, and a working computer program (Snob) that gives useful results in many different areas of science, a complete research project. The Foundation Chair of Computer Science at Monash University, Chris is also particularly remembered for his work on the Wallace multiplier [85, 86], pseudorandom number generators [14, 89], and operating systems [6, 99]. MML was developed [91, 92] in practical and theoretical directions and was applied to many inference problems by Chris, co-workers, postgraduates, and postdocs. One of my personal favourite applications is Jon Patrick s modelling of megalithic stone circles [65, 66]. I first heard about MML over lunch one day which led to applying it to biological sequence alignment [3] and related problems [15], and eventually after many twists and turns to protein structural alignment [17] and protein folding patterns [83]. Unfortunately much MML-based research that led to new inductive inference programs resulted in little shared software componentry. A new program tended to be written largely from scratch by a postgrad, postdoc or other researcher and did not contribute to any software library of shared parts. As such the programs embody reimplementations of standard parts. This phenomenon is not due to any special property of MML and actually seems to be quite common in research but it is rather ironic because, what with the complexity of models and of data being measured in the same units, MML is well suited to writing components that can be reused and supplied as parameters to other inductive inference software. The first MML book is the one written by Chris Wallace [92] but published posthumously; it is the reference work for MML theory. This other MML book is an attempt to do a combined MML and Software Engineering analysis of inductive inference software. Sound programming skills are needed to write new application vii

6 viii Preface programs for inductive inference problems. Some mathematical skills, particularly in calculus and linear algebra, are needed to do a new MML analysis of one s favourite statistical model. Melbourne, Victoria, Australia Lloyd Allison

7 Acknowledgements Chris Wallace was a great inspiration and always generous with ideas. He is sadly missed. This book was begun largely at the urging of Arun Konagurthu who at times shows aspects of both an irresistible force and an immovable object. He also contributed to the content and examples and fought valiantly in the typesetting wars. Leigh Fitzgibbon, Josh Comley, and Rodney O Donnell deserve special mention for contributions [18, 34, 61] to early attempts to create and use general MML software. Many thanks go to Dianna Kenny for sharing her data on musicians and mortality (Sect. 7.5). I am indebted to those who read parts, a little or a lot, of drafts of the book and who suggested improvements, a few or many, in alphabetical order: Rohan Baxter, Minh Duc Cao, Trevor Dix, Rodney O Donnell, Arun Konagurthu, Francois Petitjean, Joel Reicher, Daniel Schmidt. But, as they say, the mistakes are all my own. ix

8 Contents Preface... vii Acknowledgements... ix 1 Introduction Explanation versus Prediction Models Implementation Estimators Information MML MML Single Continuous Parameter θ Multiple Continuous Parameters Outline Discrete Uniform Two State MultiState Adaptive Integers Universal Codes Wallace Tree Code A Warning Geometric Poisson Shifting xi

9 xii Contents 4 Continuous Uniform Exponential Normal Normal Given μ Laplace Comparing Models Function-Models Konstant Function-Model, K Multinomial Intervals Conditional Probability Table (CPT) Multivariate Independent Dependent Data Operations Mixture Models The Message Search Implementation An Example Mixture Model The 27 Club Function-Models Naive Bayes An Example Note: Not So Naive Classification Trees The Message Search Implementation An Example Missing Values Vectors D-Dimensions, R D Implementation Norm and Direction... 91

10 Contents xiii 9.2 Simplex and x 1 = Uniform Implementation Dirichlet Directions in R D Uniform von Mises Fisher (vmf) Distribution Linear Regression Single Variable Implementation Single Dependence Unknown Single Dependence Multiple Input Variables Implementation Graphs Ordered or Unordered Graphs Adjacency Matrix v. Adjacency Lists Models of Graphs Gilbert, Erdos and Renyi Models Gilbert, Erdos and Renyi Adaptive Skewed Degree Model Motif Models A Simple Motif Model An Adaptive Motif Model Comparisons Biological Networks Bits and Pieces Priors Parameterisation Estimators Data Accuracy of Measurement (AoM) Small or Big Data Data Compression Techniques Getting Started Testing and Evaluating Results Evaluating Function-Models Programming Languages logsum Data-Sets

11 xiv Contents 13 An Implementation Support Values Utilities Maths Vectors Graphs Models Example Programs Glossary Bibliography Index

International Series in Operations Research & Management Science

International Series in Operations Research & Management Science Volume 240 Series Editor Camille C. Price Stephen F. Austin State University, TX, USA Associate Series Editor Joe Zhu Worcester Polytechnic