Dedicated to the Memory of Sir Harold Jeffreys, who saw the truth and preserved it.
Fragmentary Edition of June 1994
General comments (BY OTHERS, NOT E.T. Jaynes') about the book and maxent in general.
Chapter 1 Plausible Reasoning Chapter 2 Quantitative Rules: The Cox Theorems Fig. 2-1 Chapter 3 Elementary Sampling Theory Chapter 4 Elementary Hypothesis Testing Fig. 4-1 Chapter 5 Queer Uses for Probability Theory Chapter 6 Elementary Parameter Estimation Fig. 6-1 Fig. 6-2 Chapter 7 The Central Gaussian, or Normal, Distribution Chapter 8 Sufficiency, Ancillarity, and All That Chapter 9 Repetitive Experiments: Probability and Frequency Chapter 10 Physics of ``Random Experiments'' Chapter 11 The Entropy Principle Chapter 12 Ignorance Priors -- Transformation Groups Chapter 13 Decision Theory: Historical Survey Chapter 14 Simple Applications of Decision Theory Chapter 15 Paradoxes of Probability Theory Fig. 15-1 Chapter 16 Orthodox Statistics: Historical Background Chapter 17 Principles and Pathology of Orthodox Statistics Chapter 18 The A --Distribution and Rule of Succession
Chapter 19 Physical Measurements Chapter 20 Regression and Linear Models Chapter 21 Estimation with Cauchy and t--Distributions Chapter 22 Time Series Analysis and Autoregressive Models Chapter 23 Spectrum / Shape Analysis Chapter 24 Model Comparison and Robustness Chapter 25 Image Reconstruction Chapter 26 Marginalization Theory Chapter 27 Communication Theory Chapter 28 Optimal Antenna and Filter Design Chapter 29 Statistical Mechanics Chapter 30 Conclusions
Appendix A Other Approaches to Probability Theory Appendix B Formalities and Mathematical Style Appendix C Convolutions and Cumulants Appendix D Dirichlet Integrals and Generating Functions Appendix E The Binomial -- Gaussian Hierarchy of Distributions Appendix F Fourier Analysis Appendix G Infinite Series Appendix H Matrix Analysis and Computation Appendix I Computer Programs
Long Contents PART A -- PRINCIPLES and ELEMENTARY APPLICATIONS Chapter 1 PLAUSIBLE REASONING Deductive and Plausible Reasoning 101 Analogies with Physical Theories 103 The Thinking Computer 104 Introducing the Robot 105 Boolean Algebra 106 Adequate Sets of Operations 108 The Basic Desiderata 111 COMMENTS 113 Common Language vs. Formal Logic 114 Nitpicking 116 Chapter 2 THE QUANTITATIVE RULES The Product Rule 201 The Sum Rule 206 Qualitative Properties 210 Numerical Values 212 Notation and Finite Sets Policy 217 COMMENTS 218 ``Subjective'' vs. ``Objective'' 218 G Theorem 218 Venn Diagrams 220 The ``Kolmogorov Axioms'' 222 Chapter 3 ELEMENTARY SAMPLING THEORY Sampling Without Replacement 301 Logic Versus Propensity 308 Reasoning from Less Precise Information 311 Expectations 313 Other Forms and Extensions 314 Probability as a Mathematical Tool 315 The Binomial Distribution 315 Sampling With Replacement 318 Digression: A Sermon on Reality vs. Models 318 Correction for Correlations 320 Simplification 325 COMMENTS 326 A Look Ahead 328 Chapter 4 ELEMENTARY HYPOTHESIS TESTING Prior Probabilities 401 Testing Binary Hypotheses with Binary Data 404 Non-Extensibility Beyond the Binary Case 409 Multiple Hypothesis Testing 411 Continuous Probability Distributions (pdf's) 418 Testing an Infinite Number of Hypotheses 419 Simple and Compound (or Composite) Hypotheses 424 COMMENTS 425 Etymology 425 What Have We Accomplished? 426 Chapter 5 QUEER USES FOR PROBABILITY THEORY Extrasensory Perception 501 Mrs. Stewart's Telepathic Powers 502 Converging and Diverging Views 507 Visual Perception 511 The Discovery of Neptune 512 Digression on Alternative Hypotheses 514 Horseracing and Weather Forecasting 517 Paradoxes of Intuition 520 Bayesian Jurisprudence 521 COMMENTS 522 CONTENTS CONTENTS Chapter 6 ELEMENTARY PARAMETER ESTIMATION Inversion of the Urn Distributions 601 Both N and R Unknown 601 Uniform Prior 604 Truncated Uniform Priors 608 A Concave Prior 609 The Binomial Monkey Prior 611 Metamorphosis into Continuous Parameter Estimation 613 Estimation with a Binomial Sampling Distribution 614 Digression on Optional Stopping 616 The Likelihood Principle 617 Compound Estimation Problems 617 A Simple Bayesian Estimate: Quantitative Prior Information 618 From Posterior Distribution to Estimate 621 Back to the Problem 624 Effects of Qualitative Prior Information 626 The Jeffreys Prior 629 The Point of it All 630 Interval Estimation 632 Calculation of Variance 632 Generalization and Asymptotic Forms 634 A More Careful Asymptotic Derivation 635 COMMENTS 636 Chapter 7 THE CENTRAL GAUSSIAN, OR NORMAL DISTRIBUTION The Gravitating Phenomenon 701 The Herschel--Maxwell Derivation 702 The Gauss Derivation 703 Historical Importance of Gauss' Result 704 The Landon Derivation 705 Why the Ubiquitous Use of Gaussian Distributions? 707 Why the Ubiquitous Success? 709 The Near--Irrelevance of Sampling Distributions 711 The Remarkable Efficiency of Information Transfer 712 Nuisance Parameters as Safety Devices 713 More General Properties 714 Convolution of Gaussians 715 Galton's Discovery 715 Population Dynamics and Darwinian Evolution 717 Resolution of Distributions into Gaussians 719 The Central Limit Theorem 722 Accuracy of Computations 723 COMMENTS 724 Terminology Again 724 The Great Inequality of Jupiter and Saturn 726 Chapter 8 SUFFICIENCY, ANCILLARITY, AND ALL THAT Sufficiency 801 Fisher Sufficiency 803 Generalized Sufficiency 804 Examples Sufficiency Plus Nuisance Parameters The Pitman--Koopman Theorem The Likelihood Principle Effect of Nuisance Parameters Use of Ancillary Information Relation to the Likelihood Principle Asymptotic Likelihood: Fisher Information Combining Evidence from Different Sources: Meta--Analysis Pooling the Data Fine--Grained Propositions: Sam's Broken Thermometer COMMENTS The Fallacy of Sample Re--use A Folk--Theorem Effect of Prior Information Clever Tricks and Gamesmanship Chapter 9 REPETITIVE EXPERIMENTS -- PROBABILITY AND FREQUENCY Physical Experiments 901 The Poorly Informed Robot 902 Induction 905 Partition Function Algorithms 907 Relation to Generating Functions 911 Another Way of Looking At It 912 Probability and Frequency 913 Halley's Mortality Table 915 COMMENTS: The Irrationalists 918 Chapter 10 PHYSICS OF ``RANDOM EXPERIMENTS'' An Interesting Correlation 1001 Historical Background 1002 How to Cheat at Coin and Die Tossing 1003 Experimental Evidence 1006 Bridge Hands 1007 General Random Experiments 1008 Induction Revisited 1010 But What About Quantum Theory? 1011 Mechanics Under the Clouds 1012 More on Coins and Symmetry 1013 Independence of Tosses 1017 The Arrogance of the Uninformed 1019 Chapter 11 DISCRETE PRIOR PROBABILITIES~--~THE ENTROPY PRINCIPLE A New Kind of Prior Information 1101 Minimum 1103 Entropy: Shannon's Theorem 1104 The Wallis Derivation 1108 An Example 1110 Generalization: A More Rigorous Proof 1111 Formal Properties of Maximum Entropy Distributions 1113 Conceptual Problems: Frequency Correspondence 1120 COMMENTS 1124 Chapter 12 UNINFORMATIVE PRIORS~--~TRANSFORMATION GROUPS Chapter 13 DECISION THEORY~--~HISTORICAL BACKGROUND Inference vs. Decision 1301 Daniel Bernoulli's Suggestion 1302 The Rationale of Insurance 1303 Entropy and Utility 1305 The Honest Weatherman 1305 Reactions to Daniel Bernoulli and Laplace 1306 Wald's Decision Theory 1307 Parameter Estimation for Minimum Loss 1310 Reformulation of the Problem 1312 Effect of Varying Loss Functions 1315 General Decision Theory 1316 COMMENTS 1317 ``Objectivity'' of Decision Theory 1317 Loss Functions in Human Society 1319 A New Look at the Jeffreys Prior 1320 Decision Theory is not Fundamental 1320 Another Dimension? 1321 Chapter 14 SIMPLE APPLICATIONS OF DECISION THEORY Definitions and Preliminaries 1401 Sufficiency and Information 1403 Loss Functions and Criteria of Optimal Performance 1404 A Discrete Example 1406 How Would Our Robot Do It? 1410 Historical Remarks 1411 The Widget Problem 1412 Solution for Stage 2 1414 Solution for Stage 3 1416 Solution for Stage 4 Chapter 15 PARADOXES OF PROBABILITY THEORY How Do Paradoxes Survive and Grow? 1501 Summing a Series the Easy Way 1502 Nonconglomerability 1503 Strong Inconsistency 1505 Finite vs. Countable Additivity 1511 The Borel--Kolmogorov Paradox 1513 The Marginalization Paradox 1516 How to Mass--produce Paradoxes 1517 COMMENTS 1518 Counting Infinite Sets? 1520 The Hausdorff Sphere Paradox 1521 Chapter 16 ORTHODOX STATISTICS -- HISTORICAL BACKGROUND The Early Problems 1601 Sociology of Orthodox Statistics 1602 Ronald Fisher, Harold Jeffreys, and Jerzy Neyman 1603 Pre--data and Post--data Considerations 1608 The Sampling Distribution for an Estimator 1609 Pro--causal and Anti--Causal Bias 1611 What is Real; the Probability or the Phenomenon? 1613 COMMENTS 1613 Chapter 17 PRINCIPLES AND PATHOLOGY OF ORTHODOX STATISTICS Unbiased Estimators Confidence Intervals Nuisance Parameters Ancillary Statistics Significance Tests The Weather in Central Park More Communication Difficulties How Can This Be? Probability Theory is Different COMMENTS Gamesmanship What Does `Bayesian' Mean? Chapter 18 THE A --DISTRIBUTION AND RULE OF SUCCESSION Memory Storage for Old Robots 1801 Relevance 1803 A Surprising Consequence 1804 An Application 1806 Laplace's Rule of Succession 1808 Jeffreys' Objection 1810 Bass or Carp? 1811 So Where Does This Leave The Rule? 1811 Generalization 1812 Confirmation and Weight of Evidence 1815 Carnap's Inductive Methods 1817 PART B - ADVANCED APPLICATIONS Chapter 19 PHYSICAL MEASUREMENTS Reduction of Equations of Condition 1901 Reformulation as a Decision Problem 1903 Sermon on Gaussian Error Distributions 1904 The Underdetermined Case: K is Singular 1906 The Overdetermined Case: K Can be Made Nonsingular 1906 Numerical Evaluation of the Result 1907 Accuracy of the Estimates 1909 COMMENTS: a Paradox 1910 Chapter 20 REGRESSION AND LINEAR MODELS Chapter 21 ESTIMATION WITH CAUCHY AND t--DISTRIBUTIONS Chapter 22 TIME SERIES ANALYSIS AND AUTOREGRESSIVE MODELS Chapter 23 SPECTRUM / SHAPE ANALYSIS Chapter 24 MODEL COMPARISON AND ROBUSTNESS The Bayesian Basis of it All 2401 The Occam Factors 2402 Chapter 25 MARGINALIZATION THEORY Chapter 26 IMAGE RECONSTRUCTION Chapter 27 COMMUNICATION THEORY Origins of the Theory 2701 The Noiseless Channel 2702 The Information Source 2706 Does the English Language Have Statistical Properties? 2708 Optimum Encoding: Letter Frequencies Known 2709 Better Encoding from Knowledge of Digram Frequencies 2712 Relation to a Stochastic Model 2715 The Noisy Channel 2718 Fixing a Noisy Channel: the Checksum Algorithm 2718 Chapter 28 OPTIMAL ANTENNA AND FILTER DESIGN Chapter 29 STATISTICAL MECHANICS Chapter 30 CONCLUSIONS APPENDICES Appendix A Other Approaches to Probability Theory The Kolmogorov System of Probability A 1 The de Finetti System of Probability A 5 Comparative Probability A 6 Holdouts Against Comparability A 7 Speculations About Lattice Theories A 8 Appendix B Formalities and Mathematical Style Notation and Logical Hierarchy B 1 Our ``Cautious Approach" Policy B 3 Willy Feller on Measure Theory B 3 Kronecker vs. Weierstrasz B 5 What is a Legitimate Mathematical Function? B 6 Nondifferentiable Functions B 8 What am I Supposed to Publish? B 10 Mathematical Courtesy B 11 Appendix C Convolutions and Cumulants Relation of Cumulants and Moments C 4 Examples C 5 Appendix D Dirichlet Integrals and Generating Functions Appendix E The Binomial~--~Gaussian Hierarchy of Distributions Appendix F Fourier Theory Appendix G Infinite Series Appendix H Matrix Analysis and Computation Appendix 3pt I Computer Programs REFERENCES NAME INDEX SUBJECT INDEX