PROBABILITY THEORY:
THE LOGIC OF SCIENCE

by
E. T. Jaynes
Wayman Crow Professor of Physics
Washington University
St. Louis, MO 63130, U.S.A.

Dedicated to the Memory of Sir Harold Jeffreys, who saw the truth and preserved it.

Fragmentary Edition of June 1994

Short Contents

PREFACE

COMMENTS

General comments (BY OTHERS, NOT E.T. Jaynes') about the book and maxent in general.

PART A - PRINCIPLES AND ELEMENTARY APPLICATIONS

Chapter  1      Plausible Reasoning
Chapter  2      Quantitative Rules: The Cox Theorems
Fig. 2-1
Chapter  3      Elementary Sampling Theory
Chapter  4      Elementary Hypothesis Testing
Fig. 4-1
Chapter  5      Queer Uses for Probability Theory
Chapter  6      Elementary Parameter Estimation
Fig. 6-1
Fig. 6-2
Chapter  7      The Central Gaussian, or Normal, Distribution
Chapter  8      Sufficiency, Ancillarity, and All That
Chapter  9      Repetitive Experiments: Probability and Frequency
Chapter 10      Physics of ``Random Experiments''
Chapter 11      The Entropy Principle
Chapter 12      Ignorance Priors -- Transformation Groups
Chapter 13      Decision Theory: Historical Survey
Chapter 14      Simple Applications of Decision Theory
Chapter 15      Paradoxes of Probability Theory
Fig. 15-1
Chapter 16      Orthodox Statistics: Historical Background
Chapter 17      Principles and Pathology of Orthodox Statistics
Chapter 18      The A --Distribution and Rule of Succession

PART B -- ADVANCED APPLICATIONS


Chapter 19      Physical Measurements
Chapter 20      Regression and Linear Models
Chapter 21      Estimation with Cauchy and  t--Distributions
Chapter 22      Time Series Analysis and Autoregressive Models
Chapter 23      Spectrum / Shape Analysis
Chapter 24      Model Comparison and Robustness
Chapter 25      Image Reconstruction
Chapter 26      Marginalization Theory
Chapter 27      Communication Theory
Chapter 28      Optimal Antenna and Filter Design
Chapter 29      Statistical Mechanics
Chapter 30      Maximum Entropy: Matrix Formulation

APPENDICES


Appendix A       Other Approaches to Probability Theory
Appendix B       Formalities and Mathematical Style
Appendix C       Convolutions and Cumulants
Appendix D       Dirichlet Integrals and Generating Functions
Appendix E       The Binomial -- Gaussian Hierarchy of Distributions
Appendix F       Fourier Analysis
Appendix G       Infinite Series
Appendix H       Matrix Analysis and Computation
Appendix I       Computer Programs

REFERENCES

List of references


     Long Contents 


   PART   A -- PRINCIPLES and ELEMENTARY   APPLICATIONS  
      

Chapter   1    PLAUSIBLE   REASONING

   
Deductive and Plausible Reasoning                          101
Analogies with Physical Theories                           103
The Thinking Computer                                      104
Introducing the Robot                                      105
Boolean Algebra                                            106
Adequate Sets of Operations                                108
The Basic Desiderata                                       111
COMMENTS                                                   113
   Common Language vs. Formal Logic               114
   Nitpicking                                     116  

  Chapter   2    THE   QUANTITATIVE   RULES  

   
The Product Rule                                            201 
The Sum Rule                                                206
Qualitative Properties                                      210
Numerical Values                                            212
Notation and Finite Sets Policy                             217
COMMENTS                                                    218
   ``Subjective'' vs. ``Objective''                218
   G  Theorem                               218
   Venn Diagrams                                   220
   The ``Kolmogorov Axioms''                       222  

  
Chapter   3    ELEMENTARY   SAMPLING   THEORY 

   
Sampling Without Replacement                                  301
Logic Versus Propensity                                       308
Reasoning from Less Precise Information                       311
Expectations                                                  313
Other Forms and Extensions                                    314
Probability as a Mathematical Tool                            315
The Binomial Distribution                                     315
Sampling With Replacement                                     318
Digression: A Sermon on Reality vs. Models                    318
Correction for Correlations                                   320
Simplification                                                325
COMMENTS                                                      326
  A Look Ahead                                           328 
 

Chapter   4    ELEMENTARY   HYPOTHESIS   TESTING 

  
Prior Probabilities                                      401
Testing Binary Hypotheses with Binary Data               404
Non-Extensibility Beyond the Binary Case                 409
Multiple Hypothesis Testing                              411
Continuous Probability Distributions (pdf's)             418
Testing an Infinite Number of Hypotheses                 419
Simple and Compound (or Composite) Hypotheses            424
COMMENTS                                                 425
   Etymology                                    425
   What Have We Accomplished?                   426 

  
Chapter   5    QUEER   USES   FOR   PROBABILITY   THEORY

   
Extrasensory Perception                                  501
Mrs. Stewart's Telepathic Powers                         502
Converging and Diverging Views                           507
Visual Perception                                        511
The Discovery of Neptune                                 512
Digression on Alternative Hypotheses                     514
Horseracing and Weather Forecasting                      517
Paradoxes of Intuition                                   520
Bayesian Jurisprudence                                   521
COMMENTS                                                 522   




     CONTENTS        CONTENTS    

  
Chapter   6   ELEMENTARY   PARAMETER   ESTIMATION 

  
Inversion of the Urn Distributions                        601
Both N and R Unknown                                      601
Uniform Prior                                             604
Truncated Uniform Priors                                  608
A Concave Prior                                           609
The Binomial Monkey Prior                                 611
Metamorphosis into Continuous Parameter Estimation        613
Estimation with a Binomial Sampling Distribution          614
Digression on Optional Stopping                           616
The Likelihood Principle                                  617
Compound Estimation Problems                              617
A Simple Bayesian Estimate: Quantitative Prior Information    618
From Posterior Distribution to Estimate                   621
Back to the Problem                                       624
Effects of Qualitative Prior Information                  626
The Jeffreys Prior                                        629
The Point of it All                                       630
Interval Estimation                                       632
Calculation of Variance                                   632
Generalization and Asymptotic Forms                       634
A More Careful Asymptotic Derivation                      635
COMMENTS                                                  636 

         
Chapter   7   THE   CENTRAL   GAUSSIAN,   OR   NORMAL   DISTRIBUTION 

  
The Gravitating Phenomenon                                  701
The Herschel--Maxwell Derivation                            702
The Gauss Derivation                                        703
Historical Importance of Gauss' Result                      704
The Landon Derivation                                       705
Why the Ubiquitous Use of Gaussian Distributions?           707
Why the Ubiquitous Success?                                 709
The Near--Irrelevance of Sampling Distributions             711
The Remarkable Efficiency of Information Transfer           712
Nuisance Parameters as Safety Devices                       713
More General Properties                                     714
Convolution of Gaussians                                    715
Galton's Discovery                                          715
Population Dynamics and Darwinian Evolution                 717
Resolution of Distributions into Gaussians                  719
The Central Limit Theorem                                   722
Accuracy of Computations                                    723
COMMENTS                                                    724
   Terminology Again                               724
   The Great Inequality of Jupiter and Saturn      726 

 
Chapter   8   SUFFICIENCY,   ANCILLARITY,   AND   ALL   THAT

  
Sufficiency                                                  801
Fisher Sufficiency                                           803
Generalized Sufficiency                                      804
Examples
Sufficiency Plus Nuisance Parameters
The Pitman--Koopman Theorem
The Likelihood Principle
Effect of Nuisance Parameters
Use of Ancillary Information
Relation to the Likelihood Principle
Asymptotic Likelihood: Fisher Information
Combining Evidence from Different Sources: Meta--Analysis
Pooling the Data
Fine--Grained Propositions: Sam's Broken Thermometer
COMMENTS
The Fallacy of Sample Re--use
A Folk--Theorem
Effect of Prior Information
Clever Tricks and Gamesmanship 

 
Chapter   9   REPETITIVE   EXPERIMENTS -- PROBABILITY   AND   FREQUENCY 

  
Physical Experiments                                       901
The Poorly Informed Robot                                  902
Induction                                                  905
Partition Function Algorithms                              907
Relation to Generating Functions                           911
Another Way of Looking At It                               912
Probability and Frequency                                  913
Halley's Mortality Table                                   915
COMMENTS: The Irrationalists                               918 

 
Chapter   10   PHYSICS   OF   ``RANDOM   EXPERIMENTS''

  
An Interesting Correlation                                   1001
Historical Background                                        1002
How to Cheat at Coin and Die Tossing                         1003
Experimental Evidence                                        1006
Bridge Hands                                                 1007
General Random Experiments                                   1008
Induction Revisited                                          1010
But What About Quantum Theory?                               1011
Mechanics Under the Clouds                                   1012
More on Coins and Symmetry                                   1013
Independence of Tosses                                       1017
The Arrogance of the Uninformed                              1019 

 
Chapter  11   DISCRETE   PRIOR   PROBABILITIES~--~THE   
ENTROPY   PRINCIPLE 

  
A New Kind of Prior Information                              1101
Minimum                                           1103
Entropy: Shannon's Theorem                                   1104
The Wallis Derivation                                        1108
An Example                                                   1110
Generalization: A More Rigorous Proof                        1111
Formal Properties of Maximum Entropy Distributions           1113
Conceptual Problems: Frequency Correspondence                1120
COMMENTS                                                     1124 

 
Chapter  12   UNINFORMATIVE   PRIORS~--~TRANSFORMATION   GROUPS 

 
Chapter 13   DECISION   THEORY~--~HISTORICAL   BACKGROUND

  
Inference vs. Decision                                      1301
Daniel Bernoulli's Suggestion                               1302
The Rationale of Insurance                                  1303
Entropy and Utility                                         1305
The Honest Weatherman                                       1305
Reactions to Daniel Bernoulli and Laplace                   1306
Wald's Decision Theory                                      1307
Parameter Estimation for Minimum Loss                       1310
Reformulation of the Problem                                1312
Effect of Varying Loss Functions                            1315
General Decision Theory                                     1316
COMMENTS                                                    1317
   ``Objectivity'' of Decision Theory              1317
   Loss Functions in Human Society                 1319
   A New Look at the Jeffreys Prior                1320
   Decision Theory is not Fundamental              1320
   Another Dimension?                              1321 

 
Chapter 14   SIMPLE   APPLICATIONS   OF   DECISION   THEORY 

  
Definitions and Preliminaries                                 1401
Sufficiency and Information                                   1403
Loss Functions and Criteria of Optimal Performance            1404
A Discrete Example                                            1406
How Would Our Robot Do It?                                    1410
Historical Remarks                                            1411
The Widget Problem                                            1412
Solution for Stage 2                                          1414
Solution for Stage 3                                          1416
Solution for Stage 4                                           

 
Chapter 15   PARADOXES   OF   PROBABILITY   THEORY 

  
How Do Paradoxes Survive and Grow?                           1501
Summing a Series the Easy Way                                1502
Nonconglomerability                                          1503
Strong Inconsistency                                         1505
Finite vs. Countable Additivity                              1511
The Borel--Kolmogorov Paradox                                1513
The Marginalization Paradox                                  1516
How to Mass--produce Paradoxes                               1517
COMMENTS                                                     1518
   Counting Infinite Sets?                          1520
   The Hausdorff Sphere Paradox                     1521 

 
Chapter 16   ORTHODOX   STATISTICS -- HISTORICAL   BACKGROUND

  
The Early Problems                                         1601
Sociology of Orthodox Statistics                           1602
Ronald Fisher, Harold Jeffreys, and Jerzy Neyman           1603
Pre--data and Post--data Considerations                    1608
The Sampling Distribution for an Estimator                 1609
Pro--causal and Anti--Causal Bias                          1611
What is Real; the Probability or the Phenomenon?           1613
COMMENTS                                                   1613 

 
Chapter   17   PRINCIPLES   AND   PATHOLOGY   OF   ORTHODOX   STATISTICS

  
Unbiased Estimators
Confidence Intervals
Nuisance Parameters
Ancillary Statistics
Significance Tests
The Weather in Central Park
More Communication Difficulties
How Can This Be?
Probability Theory is Different
COMMENTS
   Gamesmanship
   What Does `Bayesian' Mean? 

 
Chapter   18   THE   A --DISTRIBUTION   AND   RULE   OF   SUCCESSION

  
Memory Storage for Old Robots                              1801
Relevance                                                  1803
A Surprising Consequence                                   1804
An Application                                             1806
Laplace's Rule of Succession                               1808
Jeffreys' Objection                                        1810
Bass or Carp?                                              1811
So Where Does This Leave The Rule?                         1811
Generalization                                             1812
Confirmation and Weight of Evidence                        1815
Carnap's Inductive Methods                                 1817 


 
   PART B - ADVANCED   APPLICATIONS   

Chapter   19    PHYSICAL   MEASUREMENTS

  
Reduction of Equations of Condition                      1901
Reformulation as a Decision Problem                      1903
Sermon on Gaussian Error Distributions                   1904
The Underdetermined Case: K is Singular                  1906
The Overdetermined Case: K Can be Made Nonsingular       1906
Numerical Evaluation of the Result                       1907
Accuracy of the Estimates                                1909
COMMENTS: a Paradox                                      1910 

 
Chapter   20   REGRESSION AND LINEAR MODELS 

Chapter   21   ESTIMATION   WITH   CAUCHY   AND   t--DISTRIBUTIONS 


Chapter   22   TIME   SERIES   ANALYSIS   AND   AUTOREGRESSIVE   MODELS

Chapter   23   SPECTRUM / SHAPE   ANALYSIS

Chapter   24   MODEL   COMPARISON   AND   ROBUSTNESS

  
The Bayesian Basis of it All                                 2401
The Occam Factors                                            2402 

 
Chapter   25   MARGINALIZATION THEORY

Chapter   26   IMAGE   RECONSTRUCTION

Chapter   27   COMMUNICATION   THEORY

  
Origins of the Theory                                      2701
The Noiseless Channel                                      2702
The Information Source                                     2706
Does the English Language Have Statistical Properties?     2708
Optimum Encoding: Letter Frequencies Known                 2709
Better Encoding from Knowledge of Digram Frequencies       2712
Relation to a Stochastic Model                             2715
The Noisy Channel                                          2718
Fixing a Noisy Channel: the Checksum Algorithm             2718 

 
Chapter   28   OPTIMAL   ANTENNA   AND   FILTER   DESIGN

Chapter   29   STATISTICAL   MECHANICS

Chapter   30   CONCLUSIONS

 
   APPENDICES  

  
Appendix A   Other Approaches to Probability Theory 

   
The Kolmogorov System of Probability                       A 1 
The de Finetti System of Probability                       A 5
Comparative Probability                                    A 6 
Holdouts Against Comparability                             A 7
Speculations About Lattice Theories                        A 8   

 
Appendix B   Formalities and Mathematical Style

  
Notation and Logical Hierarchy                             B 1
Our ``Cautious Approach" Policy                            B 3
Willy Feller on Measure Theory                             B 3
Kronecker vs. Weierstrasz                                  B 5
What is a Legitimate Mathematical Function?                B 6
Nondifferentiable Functions                                B 8
What am I Supposed to Publish?                             B 10
Mathematical Courtesy                                      B 11 

 
Appendix C   Convolutions and Cumulants 

  
Relation of Cumulants and Moments                          C 4
Examples                                                   C 5 

Appendix D   Dirichlet Integrals and Generating Functions

Appendix E   The Binomial~--~Gaussian Hierarchy of Distributions

Appendix F   Fourier Theory 

Appendix G   Infinite Series 

Appendix H   Matrix Analysis and Computation

Appendix   3pt I   Computer Programs

REFERENCES

NAME   INDEX

SUBJECT   INDEX