A Geometric Theory of Ignorance

Carlos C. Rodríguez

Department of Mathematics and Statistics The University at Albany, SUNY Albany, NY

Abstract. This is an incomplete sketch of a theory that produces a Model and a prior on it, from observed data and other explicitly stated prior information. Such a theory shows the potential of explaining the universe around, and inside us. Such a theory is ultimately a theory of ignorance. I cry out loud: it and bit from not!.

Additional information is available online at http://omega.albany.edu:8008/ignorance

CONTENTS

Introduction 2 The Main Problem of Inference 2

The prior is half the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Ignorance Relative to M and O 3 Information Geometry Tools 3

Properties of δ-information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 δ-flat Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

All the Invariant Actions for Ignorance 6 Maximum Ignorance 7 Ignorance is Negative Free Energy 8 Ignorance is Self-Adjoint^{1} 9 Ignorant Priors on δ-convex Models 10 Ignorance on δ-flat Models 10 Ignorance on Exponential Family Models 12

(0,ν)-ignorant prior on exponential family models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Conjugacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

δ-ignorance 15 Example: (0,0)-prior For Mixtures of Gaussians 16

The Complete Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

A Geometric Theory of Ignorance  July 31, 2003  1

1 With respect to the operation of truth conjugation defined in the text

Computation of dV on M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Posterior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Gibbs Sampler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Model Selection 23 Ignorant Razors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 When M ∋ t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Automatic Model Production 29 Perelman’s Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

INTRODUCTION

Infandum, regina, iubes renovare dolorem, Troianas ut opes et lamentabile regnum cruerint Danai; quaeque ipse miserrima vidi, et quorum pars magna fui. Quis talia fando Myrmidonum Dolopumve aut duri miles Ulixi temperet a lacrimis? Et iam nox umida caelo praecipitat, suadentque cadentia sidera somnos.

THE MAIN PROBLEM OF INFERENCE

From prior information O and data D = (x_{1},x_{2},...,x_{n}) = x^{n} \underline{obtain} a model M = \{P_{θ} : θ ∈ Θ\} and a prior π = π(θ) on M.

With the necessary ingredients: (O,D,M,π) we can cook a bayesian omelet on a large enough computer and instantiate,

Inference = Bayes+ Computing

where Bayes stands for his theorem,

p(θ|D,M,O) = \frac{p(D|θ,M,O)p(θ|M,O)}{p(D|M,O)}  (1)

In what follows we simply write p = p(D|θ,M,O) for the likelihood, π = p(θ|M,O) for the prior, and Z = p(D|M,O) for the evidence.

The prior is half the problem

This paper concentrates primarily on a simplified version of the main task. We assume that a model M is available as part of the prior information O. Our problem is to find the prior distribution π over M. Hence, except for the last two sections where we consider model selection, we assume that M ∈ O.

We want to find an objective mechanism for producing prior distributions π from explicitly stated prior information O when the model M is part of O. Our only principle is honesty. We demand our π to be \underline{maximally ignorant} about everything except what is explicitly contained in O.

IGNORANCE RELATIVE TO M AND O

To proceed, we need a concept of ignorance relative to a model M and possibly extra prior information O. Our proposal is based on the trivial realization that ignorance is nothing but uncertainty about truth. A quantity designed to measure the amount of ignorance contained in a given prior π over M must depend on the location of the true distribution, in the space of all distributions for the data. For the object of ignorance is truth; We are ignorant about truth. But ignorance is not just measured by the proximity of the true distribution t(x) to the model M. A small model M far away from t is not ignorant about truth, it is very knowledgeable, but about the wrong thing. It is precisely incorrect. Our initial task is to assess the amount of ignorance in a prior distribution π over a given model M, not the quality of M itself. Choosing M will be the subject of model selection that is treated at the end of this paper.

Our intuitive concept of ignorance is at the center of a tug of war between two forces pulling it in opposite directions. On the one hand we would like our ignorant prior π over M to be as spread as possible over M. On the other hand we would like π to make the parameter θ as stochastically independent as possible from the data x. In geometric terms π should be close to the uniform distribution ω over M and also the joint distribution p(x,θ) = p(x|θ)π(θ) (or p(x)π(p) in parameter free notation) should be close to the independent model t(x)π(θ) (or t(x)π(p)). Even though the true distribution t(x) is never known completely, it is often asymptotically learnable from observed data. Clearly, in the absence of actual observations or other restrictions, so that O = {M} is all that is known, the concept of ignorance should reduce to plain uniformity over M. The other extreme, i.e., when t(x) is precisely localized by O, the ignorant prior over M should reduce to a point mass at p̂∈ M where p̂ is the projection of t on M. The intermediate case should be a compromise between uniformity over M and concentration about p̂. A trade-off between uniformity and independence.

To finish our attempt to quantify relative ignorance we need only a meaningful notion of separation between probability distributions. Thus, if D_{1} is such a measure for joint distributions of (x,θ) and D_{2} for distributions over M then, the maximum ignorant prior is defined as the π that minimizes the action,

A = α D_{1}(pπ,tπ) + D_{2}(π,ω)  (2)

where α is a positive scalar parameter that measures the uncertainty about t and effectively weights the relative importance of independence versus uniformity.

In the next section we review the basic facts from information geometry. In particular, we identify the class of all the statistically meaningful measures of separation between unnormalized probability distributions as the family of δ-deviations.

INFORMATION GEOMETRY TOOLS

Differential geometry provides a powerful and natural language for statistical inference. We collect in this section some basic facts, definitions and notation for future reference. Amari’s books are still the standard reference.

Let P be the space of all probability distributions on a given (Hausdorff) measurable space. We denote by P̃ the cone of finite positive measures on the same measurable space, i.e. p̃∈ P̃ ⟺ p̃= cp with p ∈ P and c > 0. Notice that P̃ is closed under addition and multiplication by a positive number but P is not. Recently, Zhou and Rohwer have demonstrated that there is nothing to be lost but a lot to be gained by considering objects in P̃ rather than in P. We follow here their advice and most of their notation.

A model (also known as regular statistical model or hypothesis space) M is a subset of P which is also a riemannian manifold with Fisher information as the metric. For p ∈ P̃ and δ ∈ (0,1] we denote by,

l_{δ} = l_{δ}(p) = \frac{p^{δ}}{δ}  (3)

we also define l_{0} = l = logp. We call l_{δ}(p) the δ-coordinates of p ∈ P̃. Notice that l_{δ}(p) ∈ L_{1/δ} the space of δth power finite measures defined by,

L_{1/δ} = {p^{δ}f : p ∈ P̃ and f ∈ L_{1/δ}(p)}  (4)

L_{1/δ} is a Banach space with the obvious identification of objects, p^{δ}f ≡ q^{δ}g whenever the g ∈ L_{1/δ}(q) is actually (p/q)^{δ}f. Notice that p/q is a Radon-Nikodým derivative, when it exists.

Fisher information on the whole P̃ is given for all δ ∈ [0,1] by

I(p) = (g_{ij}(p)) = (< ∂_{i},∂_{j} >) = ( ∫ p ∂_{i}l ∂_{j}l) = ( ∫ ∂_{i}l_{δ} ∂_{j}l_{1−δ} )  (5)

where g_{ij}(p) is given relative to a choice of an ordered Hilbert basis in L_{2}. Thus, ∂_{i} denotes the Gateaux derivative in the direction of the chosen ith coordinate basis in L_{2}. It is convenient to think of the infinite dimensional manifold, P̃ with the above metric, as the ambient external space for our models M. The concepts of δ-geodesic, δ-flat, and δ-convex M, as it is embedded in L_{1/δ}, are then just defined with the help of the coordinates l_{δ}(p) in the Banach space L_{1/δ}. We have these properties when the δ-coordinates form respectively a straight line, flat set, and convex set in L_{1/δ}.

We denote by dV = dV_{M} the volume form for the model M. In a given parameterization,

dV = ω(θ) = |I(θ)|^{1/2}dθ = \sqrt{detI(θ)} dθ  (6)

For δ ∈ (0,1) and p,q ∈ P̃ the δ-information-deviation (or just δ-deviation for short) between p and q is,

I_{δ}(p: q) = \frac{1}{δ(1 − δ)} ∫ [δp + (1 − δ)q − p^{δ}q^{1−δ} ]  (7)

and for δ ∈ {0,1} just take limits of the expression above to obtain the Skilling actions,

I_{1}(p: q) = I_{0}(q: p) = ∫ ( q − p + p log \frac{p}{q})  (8)

When δ = 1/2,

I_{1/2}(p: q) = 2∫ (\sqrt{p}− \sqrt{q})^{2}  (9)

is twice the square of the Hellinger distance, i.e. the familiar L_{2} distance between wave functions. Notice that to compute I_{δ}(p: q) one picks any measure r ∈ P̃ dominating both p and q (e.g. p+q is always a choice), replace p and q by their densities with respect to r, i.e. p/r and q/r and carry out the integral with respect to the chosen measure r. The final result is independent of the choice of r. Thus, I_{δ} is truly a functional of the positive measures p and q and not just their densities.

Properties of δ-information

The family of δ-deviations has a number of remarkable properties:

1. homogeneity: I_{δ}(cp: cq) = c I_{δ}(p: q) for all c > 0.

2. positivity: I_{δ}(p: q) ≥ 0 with equality iff p ≡ q.

3. duality: I_{δ}(p: q) = I_{1−δ} (q: p)

4. invariance: for any T : X ⟼→ Y with positive jacobian we have,

I_{δ}(p: q) = I_{δ}(p_{T}: q_{T})

where p_{T} = p ∘ T^{−1} is the transformed probability distribution in Y .

5. sufficiency: T : X ⟼→ Y is sufficient for discriminating probability distributions p and q (i.e. ∫ p = ∫ q = 1) iff

I_{δ}(p: q) = I_{δ}(p_{T}: q_{T})

6. uniqueness: I_{δ}(p: q) are the only functions with the above properties.

7. topological equivalence: For δ ∈ (0,1) all the I_{δ} topologies are equivalent to the Hausdorff topology of the Hellinger metric, i.e. the topology generated by I_{1/2}.

8. Taylor expansion: In a given parameterization,

I_{δ}(θ + ϵv: θ) = \frac{1}{2}g_{ij}v^{i}v^{j}ϵ^{2}+ \frac{1}{6}[Γ^{0}_{ijk} + Γ^{δ}_{kij} + Γ^{1}_{jki}]v^{i}v^{j}v^{k}ϵ^{3}+ o(ϵ^{3})

where the Christoffel symbols are given by Amari’s δ-connection,

Γ^{δ}_{ijk}= ⟨ ∇^{δ} _{∂_{i}} ∂_{j},∂_{k}⟩ = ∫ p [∂_{i}∂_{j}l + δ ∂_{i}l∂_{j}l]∂_{k}l

The Levi-Civita metric connection corresponds to the self-dual case δ = 1/2.

9. eguchi relations:

g_{ij} = −∂_{i} ˙∂_{j}I_{δ}(p: q̇)|_{p=q} Γ^{δ}_{ijk}= −∂_{i}∂_{j} ˙∂_{k}I_{δ}(p: q̇)|_{p=q}

10. generalized cosine:

I_{δ}(p: r) + I_{δ}(r: q) = I_{δ}(p: q) + ∫ (l_{δ}(p) − l_{δ}(r))(l_{1−δ} (q) − l_{1−δ} (r))

11. generalized pythagoras: If the δ-geodesic connecting p to r is orthogonal to the (1 − δ)-geodesic connecting r

to q then,

I_{δ}(p: r) + I_{δ}(r: q) = I_{δ}(p: q)

Special cases of δ-information-deviations have been discovered and re-discovered more, perhaps, than any other concept in the history of statistics. With the sole exception of δ = 1/2, the δ-deviations are not symmetric, and do not satisfy the triangular inequality so they don’t define distances, in the usual way. However, the above properties make them to be the \underline{only} statistically meaningful way of measuring separation in the extended space of unnormalized probability distributions. The δ-information-deviations are the one and only one measures that are positive definite, and preserve all the fundamental symmetries of statistical inference, i.e. invariant under coordinate transformations of both the data and the parameter space and invariant under sufficient reductions of the data. If further more, one demands additivity over independent sources of information then, the only survivors are the Skilling actions which coincide with the Kullback numbers for normalized probability measures.

δ-flat Models

To finalize our quick over-view of IG we mention that intrinsically δ-flat models (i.e. with zero riemman tensor associated to the δ-connection) are always also (1 − δ)-flat and therefore admit mutually orthogonal geodesic coordinate systems (θ,η). The θ coordinates are δ-affine, i.e. make the δ-Christoffel symbols to be 0. The η coordinates are (1 − δ)-affine and make the (1 − δ)-Christoffel symbols 0. More over, in these special coordinate systems the metric g_{ij}, and its inverse g^{ij} are obtained by differentiating scalar potentials ψ(θ) and ϕ(η), i.e.,

g_{ij}(θ) = ∂_{i}∂_{j}ψ(θ), g^{ij}(η) = ∂^{i}∂^{j}ϕ(η).  (10)

and the two potentials are Legendre transforms of each other (just like entropy and free energy in Statistical Mechanics),

ψ(θ) = θ^{i}η_{i}− ϕ(η)  (11)

with

θ^{i} = ∂^{i}ϕ(η), and η_{i} = ∂_{i}ψ(θ).  (12)

In these coordinates,

I_{δ}(p_{θ} : q_{η}) = ψ(θ) + ϕ(η) − θ^{i}η_{i}.  (13)

The extended space P̃ is δ-flat and δ-convex for all δ ∈ [0,1]. The space of all the discrete distributions on a finite set of atoms is also intrinsically δ-flat for all δ. Exponential family models are intrinsically (0,1)-flat.

ALL THE INVARIANT ACTIONS FOR IGNORANCE

As shown in the previous section, Information Geometry provides the needed class of measures of separation to be used in (2). Two are needed since we need to measure separation in two different spaces. Pick I_{δ} for deviations between joint distributions of (x,θ) and pick I_{1−ν} for quantifying deviations of distributions over M. We arrive to a three parameter family of invariant actions,

A (π;α,δ,ν) = α I_{δ}(pπ : tπ) + I_{1−ν} (π : ω)  (14)

where the parameters, α > 0, δ ∈ [0,1] and ν ∈ [0,1] are free for now. The actions defined in (14) also depend on t and implicitly on M and O but it is not shown explicitly to keep the notation simple. Even though p appears on the right hand side of (14), it is integrated over inside I_{δ} so A itself is not a function of p.

The δ-deviations are well-defined in the extended space P̃ of unnormalized positive measures and the extra freedom makes the optimization of actions of type (14) simpler. To take advantage of this benefit we adhere from now on to the following notational convention:

Assumption 1 Distributions over M, like π and ω may appear unnormalized. Equations such us, π(θ) = f(θ) could mean π(θ) ∝ f(θ) when needed and π may stand for a scalar density or a form depending on the context.

We are now ready to formally define relative ignorance.

Definition 1 (relative (δ,ν)-ignorance) We say that a π^{∗} is (δ,ν)-ignorant at level α > 0 relative to M and O iff

π^{∗} = argmin_{π∈O}A (π;α,δ,ν)  (15)

where, for lack of better notation, we write π ∈ O to mean that the minimum is to be taken over all the π that agree with the prior information contained in O. Before writing the general solution for the case O = {M}, we consider the following:

Theorem 1 I_{δ}(pπ : tπ) = ∫_{M}I_{δ}(p: t) π(p)dV.

Proof Just an application of Fubini’s theorem. The case δ ∈ {0,1} follows by continuity from the case δ ∈ (0,1) for which,

δ(1 − δ)I_{δ}(pπ : tπ) = ∫ [δpπ + (1 − δ)tπ − p^{δ}π^{δ}t^{1−δ} π^{1−δ} ] = δ(1 − δ) ∫ π I_{δ}(p: t)

Q.E.D.

In words: The δ-deviation between the joint distribution p(x)π(p) and the independent model t(x)π(p) is always equal to the expected δ-deviation between p(x) and t(x), where the expectation is taken with respect to the prior π(p). Thus, the choice of δ is effectively a choice of distance for distributions over the data space. The choice of ν specifies the distance for distributions over M or, equivalently, the parameter space. With the help of this theorem let us re-write the actions (14) as,

A (π;α,δ,ν) = α ⟨I_{δ}(p: t)⟩_{π} + I_{1−ν} (π : ω)  (16)
= α \{ I_{δ}(pπ : tπ) + \frac{1}{α} I_{1−ν} (π : ω)\}  (17)

Using the standard arguments from optimization subject to inequality constraints (Khun-Tucker) we can show that the π^{∗} minimizing (16) is the same as the π^{∗} closer to the uniform over M (in the (1 − ν)-deviation) subject to a maximum average distance (in the δ-deviation) from t. In the case ν = 0 this is literally maximum entropy subject to a constraint. In a similar way, by using (17) we can choose to optimize independence subject to an inequality constraint on uniformity. We write it as a theorem,

Theorem 2 The following are all equivalent:

1. π^{∗} is (δ,ν)-ignorant at level α > 0.

2.

π^{∗} = argmax {−I_{1−ν} (π : ω)} s.t. < I_{δ}(p: t) >_{π} ≤ E_{max}  (18)

3.

π^{∗} = argmin I_{δ}(pπ : tπ) s.t. I_{ν}(ω : π) ≤ C_{max}  (19)

where E_{max},C_{max}, and α are functions of each other.

Theorem (2) provides a simple geometric characterization for our concept of ignorance. The ignorant prior is the one that maximizes spread over M subject to keeping the mean distance from a fix truth t less than a maximum acceptable value E_{max}. Clearly, if t ∉ M then E_{max}≥ I_{δ}(p̂: t) > 0 otherwise there will be no solution.

MAXIMUM IGNORANCE

We have found a path to ignorance so let us dare to walk the walk.

Theorem 3 Let

α_{min}= inf \{α > 0 : Z(α) = ∫_{M} [1 + αν I_{δ}(p: t)]^{− \frac{1}{ν}} dV < ∞\}  (20)

Then, for α > α_{min}, when O = {M} the (δ,ν)-ignorant prior at level Z^{ν}α is,

\frac{π^{∗}}{ω} = \frac{1}{Z(α)} [1 + ανI_{δ}] ^{− \frac{1}{ν}}  (21)

Proof Impose the normalization constraint and use (16) to write, A + (λ/ν)∫ π = ∫ L with,

L = Z^{ν}αI_{δ}π + \frac{π}{ν} + \frac{ω}{1 − ν} − \frac{π^{1−ν}}{1 − ν} \frac{ω^{ν}}{ν} + \frac{λ}{ν} π

where we have replaced α with Z^{ν}α. Thus,

\frac{δL}{δπ} = 0 ⟺ \frac{π^{∗}}{ω} = [1 + λ + Z^{ν}ανI_{δ}]^{− \frac{1}{ν}}

which coincides with the normalized density (21) when λ = Z^{ν} − 1. Q.E.D.

At this point the reader is encouraged to praise the author for his choice of notation. All the symbols in (21) are statistically and geometrically meaning-full. The left hand side of (21) is a Radon-Nikodým derivative. The right hand side is a normalized scalar density field over M. Several families of prior distributions that are routinely used in practical applications, as subjective priors, are special cases of (21). Hence, equation (21) may provide an objective justification for their practical use. In particular, equation (21) includes the family of multivariate Student-t distributions when e.g. M is a location model in the exponential family, δ ∈ {0,1} and ν ∈ (0,1]. For this reason we may think of (21) as a generalized-t family. The case ν ∈ {0,1} is obtained from (21) by taking limits. As ν → 0, we obtain (π^{∗}/ω) → exp(−αI_{δ}). Thus, equation (21) includes all the entropic priors as the special case ν = 0 and δ ∈ {0,1}. The (0,0)-ignorant priors include the standard^{2} family of conjugate priors for the exponential family likelihoods (see below). The case ν = 1 can be thought as a generalized multivariate Cauchy. Jeffreys’ priors are the case α = 0 but theorem (2) (see the paragraph right after it) shows that in some cases there may be an α_{min}> 0 beyond which the solution is meaningless. An α_{min} > 0 may be needed even if t ∈ M for some models with infinite volume or with unbounded densities, in order to assure that the prior is normalizable. All the traditional maximum entropy distributions are also included as the limit α → ∞.

IGNORANCE IS NEGATIVE FREE ENERGY

Let Z = Z(α,δ,ν,t) be the normalizing partition function for the family of ignorant priors (21), i.e.,

Z = ∫_{M} [1 + αν I_{δ}(p: t)]^{− \frac{1}{ν}} dV.  (22)

2 Only if they are interpreted as scalar densities on M

Definition 2 (Free Energy) The function F = F (α,δ,ν,t) given by

F = −logZ  (23)

is the free energy associated to the (δ,ν)-ignorant prior (21).

We show that the actual minimal value of the action at the optimal (δ,ν)-ignorant prior is an increasing function of the free energy and it is exactly the free energy when ν = 0.

Theorem 4 Let the model M and the ignorant prior be normalized, i.e., ∫ ω = ∫ π^{∗} = 1. Then the minimal value of the action that (21) minimizes is A^{∗} given by,

A^{∗} = \frac{1}{ν}(1 − e^{−νF} ) + νI_{1−ν}  (24)

Moreover,

lim_{ν→0}A^{∗} = F  (25)
lim_{ν→1}A^{∗} = 1 − e^{−F} + I_{0}  (26)

(27)

Proof From theorem (3) we have,

A^{∗} = Z^{ν}α < I_{δ} > +I_{1−ν}  (28)

using (21) and the normalization assumptions we compute,

I_{1−ν} = \frac{1}{ν(1 − ν)} ( 1 − ∫ ( \frac{π^{∗}}{ω} ) ^{1−ν} ω)

= \frac{1}{ν(1 − ν)} − \frac{Z^{ν}}{ν(1 − ν)} ∫ [1 + ανI_{δ}]π^{∗}
= \frac{1}{ν(1 − ν)} − \frac{Z^{ν}}{ν(1 − ν)} (1 + αν < I_{δ} >)  (29)

substituting (29) into (28),

A^{∗} = \frac{1}{ν(1 − ν)} \{1 − (1 + ν^{2}α < I_{δ} >)e^{−νF} \}
= \frac{1}{ν(1 − ν)} \{1 − (1 + ν^{2}[A^{∗}− I_{1−ν} ]e^{νF})e^{−νF} \}

solving for A^{∗} we obtain (24). Q.E.D. Theorem (4) establishes a connection between amount of ignorance and free energy that naturally prompts us to ask new questions that theorem (4) is unable to answer. For example, (24) suggests, but it does not prove, that the optimal action for ignorance A^{∗} is always an strictly increasing function of F for all ν ∈ [0,1]. We can only deduce from (24) and (25) that A^{∗} is increasing in F for ν ∈ [0,ϵ] for some ϵ > 0. A more detailed analysis of I_{1−ν} as a function of F seems to support the conjecture that A^{∗} is increasing in F for all ν ∈ [0,1]. A final proof is not available at the moment. No proof is really needed if we change the problem a little... if the mountain doesn’t come to Mohammed...

The problem, I believe, is our original definition for free energy (23). Let us look into the explanation of the traditional thermodynamic quantity by means of statistical mechanics. The phenomenological concept of free energy A = E −T S, as originally defined by Helmholtz, is related to the statistical mechanical concept of sum-over-states (i.e. partition function) Z by the usual formula A ≡ −kT logZ but only when Z = ∑_{n}e^{−E_{n}/kT} i.e. only when the partition function is the normalizing constant of a model in the exponential family. But our family of (δ,ν))-ignorant priors is not in the exponential family unless ν = 0. This suggests redefining free energy as the optimal value of our action for ignorance, in which case, ignorance is negative free energy by definition!

It is intuitively clear that ignorance is the other side of the coin of information. We should expect something like

Information + Ignorance = O.  (30)

We know from Shannon that negative entropy is information. Theorem (4) shows that, at least for ν = 0, negative free energy is ignorance. Should we then expect the sum of the two to give the whole glass of knowledge?. What is the whole glass anyway?. Shouldn’t the whole glass change somehow dynamically? For otherwise how are we going to be able to account for true evolutionary innovation?

IGNORANCE IS SELF-ADJOINT^{3}

Let us denote by π(p|t) the right hand side of (21). If we assume that t ∈ M then we can regard π(p|t) as a function of two arbitrary distributions in M. Let us also consider an arbitrary smooth relabeling of the elements of M. Thus, each distribution in M has two names, p = p(q) and q = q(p). Also, each distribution in M has two roles that it can play. It can act as the truth t or as just p. We denote the role-flipping operator with a hat on top of the name. Hence,

π(p|q̂) = [1 + ανI_{δ}(p: q̂)]^{− \frac{1}{ν}}  (31)

denotes the (δ,ν)-ignorant prior in the p-coordinates given truth q̂ represented in the q-coordinates. Since ignorant priors are scalar functions which are invariant under relabelings we have,

π(p|q̂) = π(q|p̂).  (32)

Define the operation of coordinate transposition,

(p,q) → (q,p)  (33)

that defines the transpose of π as,

π^{⊤} (p|q) = π(q|p)  (34)

and define also the operation of truth conjugation,

p → p̂  (35)

that defines the conjugate of π as,

π̄(p|q̂) = π(p̂|q).  (36)

If we define the adjoint of π as its conjugate transpose and we denote it by π^{†}, then using (32) we obtain,

π(p|q̂) = π^{⊤} (q̂|p) = π^{†}(q|p̂) = π(q|p̂).  (37)

So π = π^{†} which is our definition of self-adjoint.

3 WITH RESPECT TO THE OPERATION OF TRUTH CONJUGATION DEFINED IN THE TEXT

IGNORANT PRIORS ON δ-CONVEX MODELS

A δ-convex set of unnormalized distributions is defined, as most δ-properties are, by looking at its δ-coordinates. It is δ-convex when the set of the δ-coordinates of its members is convex as a subset of the Banach space L_{1/δ}. Straight lines of δ-coordinates are δ-geodesics so a set S is δ-convex iff given two points in S, the delta geodesics connecting them are included in S. Models which are δ-convex, have the remarkable property of being able to represent truth faithfully with one of their members. But in general there is a price to be paid. A model M always entails a compromise between simplicity (small, computationally tractable) and accuracy (big, complex). There is in general no warranty that the true distribution t(x) is in M. As the following theorem shows, when the model is δ-convex, we can replace t ∉ M by its (1 − δ)-projection q̂∈ M without missing a bit.

Theorem 5 Let M be closed (in the Hellinger topology) and δ-convex. Let π(p|t,α) denote the (δ,ν)-ignorant prior at level α when the true distribution is t. Then,

π(p|t,α) = π(p|q̂,α̂)  (38)

where p̂is the unique (1 − δ)-projection of t on M, i.e.,

I_{1−δ} (t : q̂) = min_{p∈M}I_{1−δ} (t : p)  (39)

and

α̂ = \frac{α}{1 + ανI_{δ}(q̂: t)}  (40)

Proof Use the generalized law of cosines and generalized pythagoras to show that the (1 − δ)-projection of t on a closed δ-convex M, exists, it is unique, and belongs to M. Just as it is the case in a Hilbert space (see Amari85 p99 theorem3.9). Now recall that P̃ is δ-flat for all δ so the (1 − δ)-geodesic in P̃ connecting t to the projection q̂∈ M is orthogonal to the δ-geodesic connecting q̂to an arbitrary p ∈ M. Thus, by generalized pythagoras,

I_{δ}(p: t) = I_{δ}(p: q̂) + I_{δ}(q̂: t).  (41)

Replacing (41) in the formula for the ignorant prior (21) and sticking to assumption (1) we have,

π(p|t,α) = [1 + αν I_{δ}(q̂: t) + αν I_{δ}(p: q̂)] ^{− \frac{1}{ν}}
= [1 + αν I_{δ}(q̂: t)]^{− \frac{1}{ν}} [1 + α̂ν I_{δ}(p: q̂)]^{− \frac{1}{ν}}
= π(p|q̂,α̂)  (42)

Q.E.D.

Equation (40) is interesting. It quantifies the sticker price to be paid when replacing t ∉ M with q̂∈ M. It is clear that α measures, the amount of information in the ignorant prior. The smaller the α, the closer the ignorant prior is to the uniform over M. Now, equation (40) is telling us that if we want to guess t with some distribution in M we can do so. Prior and likelihood will stay the same and therefore the inferences will be exactly the same. However, the new prior needs to be less informative about the projected truth in order to be able to match the original. On the one hand equation (40) confirms the obvious: Information about t is worth less the farther away t is from M. On the other hand, equation (40) provides a precise quantification that, as far as I know, it was not known before. For example, if t ∉ M and ν > 0, then as we collect more and more information about t so that α increases, α̂ approaches monotonically from below, the number (νI_{δ}(q̂: t))^{−1} > 0. Equation (40) also tells us that ν = 0 is very special.

IGNORANCE ON δ-FLAT MODELS

Models that are intrinsically δ-flat, admit dual geodesic coordinate systems (see (11),(12), (13)), and they are obviously δ-convex. Thus, theorem (5) is applicable. We have,

Theorem 6 Under the conditions and notation used in theorem (5). If M is δ-flat then, in the δ-affine coordinates θ,

π(θ|θ̂,α̂) = [1 + α̂ν \{(ψ(θ) − ψ(θ̂)) − (θ − θ̂) · η̂\} ] ^{− \frac{1}{ν}}  (43)

and in the (1 − δ)-affine coordinates η,

π^{†}(η|η̂,α̂) = [1 + α̂ν \{(ϕ(η) − ϕ(η̂)) − θ̂· (η − η̂)\} ] ^{− \frac{1}{ν}}  (44)

where θ̂ = θ(η̂) and η̂ = η(θ̂) are the coordinates of q̂∈ M.

Proof Let p = p_{θ} and use (13) to write,

I_{δ}(p: q̂) = I_{δ}(θ : η̂)
= ψ(θ) + ϕ(η̂) − η̂· θ
= (ψ(θ) − ψ(θ̂)) − η̂· (θ − θ̂)

the last equality follows by noticing that η̂ and θ̂ are the coordinates of the same point q̂ and thus I_{δ}(θ̂ : η̂) = 0. Replacing into the ignorance prior given by theorem (5) we obtain (43). To obtain (44) just notice that I_{δ}(θ : η̂) = I_{1−δ} (θ̂: η) Q.E.D.

A simple, but useful, corollary of theorem (6) is,

Theorem 7 Under the conditions of theorem (6) we have,

π(θ|θ̂,α̂) = [1 + α̂ν ∥θ − θ̂∥ 2 ; θ̂] ^{− \frac{1}{ν}} (1 − ϵ)  (45)

where ϵ = o(∥θ − θ̂∥ 2 ; θ̂) and ∥ · ∥θ̂ denotes the norm induced by the riemannian metric at θ̂.

Proof The result follows at once by expanding ψ(θ) in a Taylor series about θ̂, using (13) and (11) and replacing into (43). Q.E.D.

The previous theorem shows that the scalar densities of ignorant priors on δ-flat models are approximated around θ̂ by multivariate Student-t distributions centered at θ̂ on the manifold M. Recall that the (unnormalized) density of a Student-t with d degrees of freedom is,

T_{d}(x) = [1 + \frac{x^{2}}{d}] ^{− \frac{(d+1)}{2}}  (46)

and matching d to ν we obtain,

d = \frac{2}{ν} − 1  (47)

The approximation (45) becomes,

[1 + α̂ν ∥θ − θ̂∥ 2 ; θ̂] ^{− \frac{1}{ν}} = [1 + \frac{α̂(2 − ν)∥θ − θ̂∥ 2 ; θ̂}{d} ] ^{− \frac{(d+1)}{2}}  (48)

The extra factor n̂= α̂(2−ν) has a simple interpretation. It is the equivalent number of virtual observations supporting the prior since it can be thought as a factor of the metric, and the metric (Fisher information) is additive over independent observations. Using (40) we obtain,

n̂= \frac{α(2 − ν)}{1 + ανI_{δ}(q̂: t)}  (49)

I find equation (49) intrinsically cool in the way it relates geometric, statistical and information concepts. It increases as ν decreases from 1 to 0 so that,

\frac{α}{1 + αI_{δ}(q̂: t)} ≤ n̂≤ 2α  (50)

In general, when t ∈ M, n̂= α(2 − ν) and n̂= α only when ν = (1 + αI_{δ}(q̂: t))^{−1} . In particular, n̂= α when t ∈ M and ν = 1 as we shall re-discover below.

As a final comment we mention that one should expect theorems (5, 6, 7) in this section to remain useful (as first order approximations) even when the models are not exactly δ-flat. A complete perturbative analysis for a general curved model M involves higher order covariant derivatives and the modifications will surely contain δ-curvature terms. I hope to be able to continue work on this problem in the near future.

IGNORANCE ON EXPONENTIAL FAMILY MODELS

Exponential family models are intrinsically 1-flat and therefore also 0-flat. However their 0-coordinates do not form a flat manifold. The situation is just like the canonical example of a cylinder imbedded in euclidean flat space. The cylinder curves but it can be smoothly unrolled so its intrinsic geometry is not different from the geometry of the plane.

The following theorem is well known. Given its practical importance we provide a complete proof. It also makes a nice concrete example of flat models and the use of the formulas introduced in the IG overview.

Theorem 8 Exponential family models are intrinsically 1-flat. The natural parameter is 0-affine and the expectation parameter is 1-affine. Their scalar potentials are the negative free energy and the negative entropy respectively.

Proof In the natural parameter θ, the normalized likelihood of an exponential family model with vector of sufficient statistics c(x) is given by,

p_{θ}(x) = e^{θ·c(x)−ψ(θ)}  (51)

where ψ(θ) is the negative free energy,

ψ(θ) = log∫ e^{c(x)·θ}dx  (52)

This is in fact the scalar potential satisfying (10) since,

g_{ij}(θ) = −E_{θ}(∂_{i}∂_{j}l) = ∂_{i}∂_{j}ψ(θ)  (53)

as it can be readily checked by taking derivatives of the 0-coordinates, i.e. the log likelihood,

l = θ · c(x)θ − ψ(θ)  (54)
∂_{i}l = c_{i}(x) − ∂_{i}ψ(θ)  (55)
∂_{i}∂_{j}l = −∂_{i}∂_{j}ψ(θ)  (56)

Taking expectations on both sides of (55) we obtain the dual 1-affine coordinates as the expectation parameter,

η_{i} = ∂_{i}ψ(θ) = E_{θ}(c_{i}(x))  (57)

The Legendre transform of (52) gives the other potential as,

ϕ(η) = θ · η − ψ(θ)  (58)
= ∫ [c(x) · θ − ψ(θ)]e^{c(x)·θ−ψ(θ)} dx
ϕ(η) = ∫ p_{θ}(x)logp_{θ}(x) dx.  (59)

We notice also that,

δ_{ik} = \frac{∂η_{i}}{∂η_{k}} = \frac{∂η_{i}}{∂θ^{j}} \frac{∂θ^{j}}{∂η_{k}} = g_{ij} \frac{∂θ^{j}}{∂η_{k}}

where we used (57) and (53) for the last equality. This shows ∂θ^{j}/∂η_{k} = g^{jk} to be the entries of the inverse. Thus, if ∂^{i} = \frac{∂}{∂η_{i}} is the tangent vector in the η-coordinates, we have

< ∂^{i},∂_{j} >=< g^{ik}∂_{k},∂_{j} >= g^{ik}g_{kj} = δ_{j}^{i}  (60)

which is a constant (either 0 or 1) and we have,

0 = ∂_{k} < ∂^{i},∂_{j} >=<∇^{1}_{k} ∂^{i},∂_{j} > + < ∂^{i},∇^{1}_{k} ∂_{j} >= g^{im}( Γ^{1}_{kmj} + Γ^{1}_{kjm})

from where we deduce that,

Γ^{1}_{kij} (θ)+ Γ^{1}_{kji} (θ) = 0.  (61)

Now we have all the ingredients to show that exponential family models are indeed intrinsically 1-flat by showing that in the θ-coordinates all the 1-connection coefficients are zero. We have,

Γ^{1}_{ijk}= E_{θ}(∂_{i}∂_{j}l∂_{k}l) + E_{θ}(∂_{i}l∂_{j}l∂_{k}l)  (62)

The first term on the right hand side of (62) is shown to be zero by just using (56). The second term is also zero as we now show,

E_{θ}(∂_{i}l∂_{j}l∂_{k}l) = ∫ ∂_{i}p_{θ}∂_{j}l∂_{k}l
= ∫ ∂_{i}p_{θ}[\frac{∂_{i}∂_{k}p_{θ}}{p_{θ}} − ∂_{j}∂_{k}l]
= ∫ ∂_{i}l∂_{j}∂_{k}p_{θ} = ∫ [c_{i}(x) − ∂_{i}ψ(θ)]∂_{j}∂_{k}p_{θ}
= ∫ c_{i}(x)∂_{j}∂_{k}p_{θ} = ∂_{j}∂_{k}∫ c_{i}p_{θ} = ∂_{i}∂_{j}∂_{k}ψ(θ)
= ∂_{i}g_{jk}(θ) = ∂_{i} < ∂_{j},∂_{k} >
= Γ^{1}_{ijk} (θ)+ Γ^{1}_{ikj} (θ)
= 0.

Where we have used (61) for the last equality. Q.E.D.

(0,ν)-ignorant prior on exponential family models

As we have seen, exponential family models are (0,1)-flat and therefore everything that was shown above, for general convex and flat models, holds in particular for members of the exponential family. Theorem (6) gives the ignorant prior for δ ∈ {0,1} and arbitrary ν ∈ [0,1] in terms of the two potentials: the negative free energy, and the negative entropy. We have

Theorem 9 The (0,ν)-ignorant prior at level α is given, for models in the exponential family, by

π(θ|η̂,α̂) = [1 + α̂ν(ψ(θ) + ϕ(η̂) − θ · η̂)]^{− \frac{1}{ν}}  (63)

the scalar potential ψ is the negative free energy (52), the dual scalar potential ϕ is the negative entropy (59), and η̂ are the 1-coordinates of p̂.

Proof (trivial).

When data x is available the natural estimate for η̂ to be used in (63) is the MLE, i.e. the θ that makes the observed x most likely. If follows from (55) that the MLE is η̂ = c(x). We have,

Theorem 10 When M is in the exponential family, the scalar posterior density based on data x and the (0,ν)-prior with η̂ = c(x) can be written as,

π(θ|x,c(x)) = \frac{e^{−s}}{[1 + α̂νs]^{1/ν}}  (64)

where s = s(θ,x) ≥ 0 defines the surface of equiprobability given by,

ψ(θ) + ϕ(c(x)) − θ · c(x) = s  (65)

Proof Use bayes, theorem (9) and assumption (1). Q.E.D. Theorem (10) shows a remarkable property of the (0,ν)-ignorant prior for exponential family models. The prior and posterior equiprobability surfaces are the same. It also suggests a simple algorithm for sampling the posterior: Pick a surface s with probability proportional to the right hand side of (64) then sample uniformly from the surface s defined by (65).

Conjugacy

Until around 1990, conjugate priors for models in the exponential family were essentially the only ones being used in multidimensional problems. The well known PCMCMC revolution changed all that.

With conjugate priors, computation of the posterior distribution reduces to plug-in formulas for the posterior parameters that involve only the sufficient statistics of the observed data. Conjugacy was invented to avoid multidimensional integration over the parameter space. Essentially to keep it simple. As we show here, these priors turn out to be ignorant provided you think of them as scalar densities on the manifold M, i.e. as densities with respect to dV_{M} (see (6)).

Theorem 11 The (0,0)-ignorant prior over a model M in the exponential family of distributions is conjugate.

Proof Exponential family models are (0,1)-flat. In the standard dual coordinates for flat models the (0,0)-ignorant prior is given by (43) when ν → 0 as,

π(θ|θ̂,α) = e^{(αθ̂·θ−αψ(θ))}  (66)

where we have used the fact that α̂ = α since ν = 0 (see (40)). On the other hand the normalized likelihood is,

p_{θ}(x) = e^{(c(x)·θ−ψ(θ))}.  (67)

Where, c(x) is the vector of sufficient statistics and,

ψ(θ) = log∫ e^{θ·c(x)}dx  (68)

Thus, the posterior is,

π(θ|x,θ̂,α) = e^{((c(x)+αθ̂)·θ−(α+1)ψ(θ))}  (69)

this is in the same family as the prior and therefore it is conjugate. Q.E.D.

δ-IGNORANCE

As we have seen, ignorance involves a trade off between uniformity and independence. In this section we show that independence by itself is enough to define a simplified notion of ignorance over a model M. Moreover, the previous notion of (δ,ν)-ignorance at level α coincides with this new version when (δ,ν) = (1,0).

The idea is straight forward. Pick one of the available measures of separation (e.g. I_{δ})on the space of joint distributions of (x,θ) and measure the δ-deviation between p(x,θ) = p(x|θ)π(θ) ≡ pπ and an arbitrary independent model t(x)η(θ) ≡ tη. The quantity I_{δ}(pπ : tη) measures closeness from the dependent to the specified independent model. We have,

Theorem 12 For normalized p and t (i.e., when ∫ p = ∫ t = 1) we have,

I_{δ}(pπ : tη) = I_{δ}(π : η) + ∫_{M}I_{δ}(p: t)π^{δ}η^{1−δ}  (70)

Proof Just write,

δ(1 − δ)I_{δ}(pπ : tη) = ∫ ∫ [δpπ + (1 − δ)tη − p^{δ}π^{δ}t^{1−δ} η^{1−δ} ]
= ∫ [δπ + (1 − δ)η − (∫ p^{δ}t^{1−δ} )π^{δ}η^{1−δ} ]  (71)
= ∫ [δπ + (1 − δ)η − π^{δ}η^{1−δ} + δ(1 − δ)I_{δ}(p: t)π^{δ}η^{1−δ} ]  (72)
= δ(1 − δ)I_{δ}(π : η) + δ(1 − δ)∫ I_{δ}(p: t)π^{δ}η^{1−δ}  (73)

where we have used the hypothesis of normalization to obtain (71) and (72). Q.E.D. If α is a positive integer we denote simply by p^{α} and t^{α} the corresponding joint distributions of α independent copies of the data vector x, i.e.,

p^{α} = ∏_{i=1}^{α} p(x_{i}|θ) and t^{α} = ∏_{i=1}^{α} t(x_{i})  (74)

Thus, for normalized p and t we have,

I_{δ}(p^{α} : t^{α}) = \frac{1}{δ(1 − δ)} ( 1 − \{ ∫ p^{δ}t^{1−δ} \}^{α} )  (75)

we use the right hand side of (75) as the definition of I_{δ}(p^{α} : t^{α}) for any α > 0. By taking limits, it follows immediately from (75) that,

I_{δ}(p^{α} : t^{α}) = α I_{δ}(p: t) for δ ∈ {0,1}  (76)

Using (70) together with (75) we define,

Definition 3 (relative δ-ignorance) We say that π^{∗} is δ-ignorant at level α > 0 relative to η,M and O iff

π^{∗} = argmin_{π∈O}I_{δ}(p^{α} π : t^{α} η)  (77)

We have,

Theorem 13 When O = {M} the δ-ignorant prior at level α relative to η is,

\frac{π^{∗}}{η} = [1 − δ(1 − δ)I_{δ}(p^{α} : t^{α})] ^{\frac{1}{1−δ}}  (78)

Proof Use theorem (12) to write the action as ∫ L then δL /δπ = 0 gives (78). Q.E.D. The following theorem is also immediate,

Theorem 14

I_{1}(p^{α} π : t^{α} ω) = A (π;α,1,0)  (79)

Proof Replace δ = 1 and ν = 0 in (16). Q.E.D. It therefore follows that,

\frac{π^{∗}}{ω} = e^{−α ∫ plog \frac{p}{t}}  (80)

is both a (1,0)-prior and a 1-prior at level α. Notice that it is also possible to switch the positions of p^{α} with t^{α} in (77) in which case the (0,1)-prior coincides with the new 0-prior at level α.

EXAMPLE: (0,0)-PRIOR FOR MIXTURES OF GAUSSIANS

The general theory presented above has many applications. To demonstrate the utility of our new understanding of statistical ignorance we work out all the details of a concrete example. In this section we compute the (0,0)-prior for the parameters of a mixture of k one dimensional gaussians. We assume k to be a given known positive integer. A simple parameterization is then given by θ = (μ,σ,ω), where μ = (μ_{1},...,μ_{k}) ∈ R^{k} is a k-dim vector of means, σ = (σ_{1},...,σ_{k}) ∈ R_{+}^{k} is a k-dim vector of standard deviations, and ω = (ω_{1},...,ω_{k}) ∈ Δ^{k−1} is the vector of mixing weights in the (k − 1)-dim simplex Δ^{k−1}. Thus, our hypothesis space M is a 3k − 1 dimensional manifold.

The Complete Likelihood

The elements of our model M are the probability distributions for the data vector (x,z) indexed by the vector of parameters θ. We assume that the complete data (x,z) is generated by first picking a label z ∈ {1,2,...,k} with probability vector ω and after that, choosing x by sampling from a gaussian with mean μ_{z} and standard deviation σ_{z}. Thus, for x ∈ R, z ∈ {1,2,...,k} and θ = (μ,σ,ω) ∈ R^{k}× R_{+}^{k} × Δ^{k−1} ≡ Θ we have,

p(x,z|θ) = p(z|θ) p(x|z,θ)
= ω_{z} \frac{1}{\sqrt{2π}σ_{z}} exp\{ −\frac{(x − μ_{z})^{2}}{2σ_{z}^{2}} \}  (81)

Hence,

M = {p(x,z|θ) : θ ∈ Θ}  (82)

Computation of dV on M

In order to obtain an ignorant prior on M, relative to the standard lebesgue measure, we need to first find the volume element of the manifold M. To use (6) we only need to find the Fisher information matrix I(θ) and its determinant |I(θ)|. We proceed in the standard way by computing first the log likelihoods l(θ), then their second derivatives ∂_{i}∂_{j}l, and finally expected values to obtain I = (g_{ij}) = (−E_{θ}(∂_{i}∂_{j}l)). The log likelihoods are obtained from (81) as,

l = l(θ) = logp(x,z|θ)
= logω_{z}− \frac{(x − μ_{z})^{2}}{2σ_{z}^{2}} − logσ_{z}− \frac{1}{2} log2π

= ∑_{j=1}^{k} \{ logω_{j} − \frac{(x − μ_{j})^{2}}{2σ_{j}^{2}} − logσ_{z}− \frac{1}{2} log2π\} 1(z = j)  (83)

where we have used the indicator function 1(z = j) which takes the value one when z = j and zero otherwise.

μ-derivatives

From (83),

\frac{∂l}{∂μ_{j}} = \frac{(x − μ_{j})}{σ_{j}^{2}} 1(z = j) and \frac{∂^{2}l}{∂μ_{j}^{2}} = −\frac{1(z = j)}{σ_{j}^{2}} =⇒ g_{μ_{j}μ_{j}} = \frac{ω_{j}}{σ_{j}^{2}}  (84)

Notice that the g_{μ,ω} terms are all zero since the first equation above does not explicitly depend on the ω_{j} parameters. We show below that the g_{μ,σ} are also zero.

σ-derivatives

Again from (83) we compute,

\frac{∂l}{∂σ_{j}} = \{ \frac{(x − μ_{j})^{2}}{σ_{j}^{3}} − \frac{1}{σ_{j}} \} 1(z = j) =⇒  \frac{∂^{2}l}{∂σ_{j}^{2}} = \{ \frac{−3(x − μ_{j})^{2}}{σ_{j}^{4}} + \frac{1}{σ_{j}^{2}} \} 1(z = j)  (85)
=⇒  g_{σ_{j}σ_{j}} = ( \frac{3}{σ_{j}^{2}} − \frac{1}{σ_{j}^{2}} ) ω_{j} = \frac{2ω_{j}}{σ_{j}^{2}}  (86)

The first equation in the line (85) does not explicitly depend on any of the ω parameters nor does it depend on any other μ_{i} except for μ_{j}. Thus, all of the off-diagonal terms, g_{σ_{j},μ_{i}} = 0 for i ≠ j and g_{σ,ω} = 0. We also have,

g_{μ_{j},σ_{j}} = E_{θ} ( \frac{∂l}{∂μ_{j}} \frac{∂l}{∂σ_{j}} )
= E_{θ} ( \frac{(x − μ_{j})^{3}}{σ_{j}^{5}} − \frac{(x − μ_{j})}{σ_{j}^{3}} )
= 0  (87)

ω-derivatives

Once more we compute using (83) and noticing that,

ω_{k} = 1 − ∑_{j=1}^{k−1} ω_{j}  (88)

and obtain,

\frac{∂l}{∂ω_{j}} = \frac{1(z = j)}{ω_{j}} − \frac{1(z = k)}{ω_{k}} for j = 1,2,...,k − 1.  (89)

Hence, it follows from (89) that all the mixed entries of type g_{ωμ} = g_{ωσ} = 0. The only non zero entries are,

g_{ω_{i}ω_{j}} = E \{ ( \frac{1(z = i)}{ω_{i}} − \frac{1(z = k)}{ω_{k}} ) ( \frac{1(z = j)}{ω_{j}} − \frac{1(z = k)}{ω_{k}} ) \}
= E \{ \frac{1(z = i)1(z = j)}{ω_{i}ω_{j}} + \frac{1(z = k)}{ω_{k}^{2}} \}

= \frac{δ_{ij}}{ω_{j}} + \frac{1}{ω_{k}}  (90)

where δ_{ij} = 1(i = j) is Kronneker’s delta.

Fisher Information Matrix

Collecting the above findings we can see that Fisher Information matrix is block diagonal,

I(θ) = [ \begin{matrix}I_{μ} & 0 & 0 \\ 0 & I_{σ} & 0 \\ 0 & 0 & I_{ω} \end{matrix} ]  (91)

where I_{μ} and I_{σ} are k × k diagonal matrices with entries,

I_{μ} = [ \frac{ω_{1}}{σ_{1}^{2}}  0 ; \frac{ω_{2}}{σ_{2}^{2}} ; ... ; 0  \frac{ω_{k}}{σ_{k}^{2}} ] and I_{σ} = [ \frac{2ω_{1}}{σ_{1}^{2}}  0 ; \frac{2ω_{2}}{σ_{2}^{2}} ; ... ; 0  \frac{2ω_{k}}{σ_{k}^{2}} ]  (92)

and I_{ω} is (k − 1) × (k − 1) with all non-zero entries,

I_{ω} = [ \begin{matrix}\frac{1}{ω_{1}} + \frac{1}{ω_{k}} & \frac{1}{ω_{k}} & \frac{1}{ω_{k}} & ··· & \frac{1}{ω_{k}} \\ \frac{1}{ω_{k}} & \frac{1}{ω_{2}} + \frac{1}{ω_{k}} & \frac{1}{ω_{k}} & ··· & \frac{1}{ω_{k}} \\ ⋮ & & ... \\ \frac{1}{ω_{k}} & & ··· & \frac{1}{ω_{k}} & \frac{1}{ω_{k−1}} + \frac{1}{ω_{k}} \end{matrix} ]  (93)

Using (6) and (91) we obtain

dV = \sqrt{|I_{μ}| |I_{σ} | |I_{ω} |}dμ dσ dω  (94)

From (92) and (93) the determinants |I_{μ}| and |I_{σ} | are just the product of their diagonal entries. The only complication that remains is the determinant |I_{ω} | that we compute as follows. First subtract the second column of (93) from the first column to obtain a matrix identical to that given by (93) except that now the first column has entries 1/ω_{1}, −1/ω_{2} and zeros down from the third to the (k − 1)st row. Call the determinant of this new (k − 1) × (k − 1) matrix D_{k−1}(ω_{1},ω_{2},...,ω_{k}). We have,

|I_{ω} | = D_{k−1}(ω_{1},ω_{2},...,ω_{k})  (95)

since the two matrices differ only by an elementary column operation that does not change the determinant. Now expanding the determinant D_{k−1} about the first column, that has all zeros except for the first two entries, we obtain the recursion,

D_{k−1}(ω_{1},...,ω_{k}) = \frac{1}{ω_{1}} D_{k−2}(ω_{2},...,ω_{k}) + (ω_{2}ω_{3}...ω_{k})^{−1}  (96)

where the second term on the right hand side of (96) is obtained by multiplying 1/ω_{2} times the determinant of the matrix obtained from (93) by erasing the first column and the second row. This co-factor is brought into an almost upper triangular matrix with main diagonal entries given by, 1/ω_{k},1/ω_{3},1/ω_{4},...,1/ω_{k−1} by subtracting the previous to last column from the last column, the second to last from the previous to last, etc, until the first column. The determinant of this almost upper triangular matrix is shown to be the product of the main diagonal by expanding about its first row that contains only zeros except for the first entry 1/ω_{k}.

From the recursion (96) we easily show,

Theorem 15 For k = 2,3,...

D_{k−1}(ω_{1},...,ω_{k}) = \frac{∑ _{j=1}^{k} ω_{j}}{∏ _{j=1}^{k} ω_{j}}  (97)

Proof By induction on k. It is true for k = 2 since,

D_{1}(ω_{1},ω_{2}) = (\frac{1}{ω_{1}} + \frac{1}{ω_{2}} ) = \frac{ω_{1}+ ω_{2}}{ω_{1}ω_{2}}  (98)

If it is true for k then using (96) and the induction hypothesis we have,

D_{k}(ω_{1},...,ω_{k+1}) = \frac{1}{ω_{1}} D_{k−1}(ω_{2},...,ω_{k+1}) + (ω_{1}··· ω_{k+1})^{−1}

= \frac{1}{ω_{1}} \frac{∑ _{j=2}^{k+1} ω_{j}}{∏ _{j=2}^{k+1} ω_{j}} + \frac{1}{∏ _{j=2}^{k+1} ω_{j}}

= \frac{∑ _{j=1}^{k+1} ω_{j}}{∏ _{j=1}^{k+1} ω_{j}}  (99)

Q.E.D. Finally, from (92), (93) and (97) we write,

dV = ∏_{j=1}^{k} \frac{ω_{j}^{1/2}}{σ_{j}^{2}} dμ dσ dω  (100)

Entropy

Assuming t ∈ M the unnormalized density (with respect to lebesgue measure dθ) of the (0,0)-prior is,

π(θ|α,θ̂) dθ = e^{−α I(θ̂:θ)} dV  (101)

where,

I(θ̂: θ) = I_{0}(θ : θ̂) = Eθ̂\{ log \frac{p(x,z|θ̂)}{p(x,z|θ)}\}  (102)

The distribution in M that represents truth has parameter,

θ̂ = (m,s,w) = (m_{1},...,m_{k},s_{1},...,s_{k},w_{1},...,w_{k})  (103)

We compute the entropy (102) by first conditioning on a value of z and then taking expectation over z, i.e.,

I(θ̂: θ) = E^{z}E^{x|z}\{ log \frac{w_{z}}{ω_{z}} + log \frac{N(x;m_{z},s_{z})}{N(x;μ_{z},σ_{z})}\}  (104)

= ∑_{j=1}^{k} w_{j} \{ log \frac{w_{j}}{ω_{j}} + log \frac{σ_{j}}{s_{j}} + \frac{(μ_{j} − m_{j})^{2}}{2σ_{j}^{2}} + \frac{s_{j}^{2}}{2σ_{j}^{2}} − \frac{1}{2}\}  (105)

where in (104) N(x;a,b) denotes the usual density for a gaussian distribution with mean a and standard deviation b. Combining (105) with the previously obtained formula for the volume element (100) we obtain,

π(θ|α,θ̂) = ∏_{j=1}^{k}  exp\{ −\frac{αw_{j}}{2σ_{j}^{2}} (μ_{j} − m_{j})^{2}\}

·σ_{j}^{−αw_{j}−2} exp\{ −\frac{αw_{j}}{2σ_{j}^{2}} s_{j}^{2}\}  (106)

·ω_{j}^{αw_{j}+ \frac{1}{2}}

Equation (106) shows, I believe correctly for the first time, the (0,0)-prior for mixtures of univariate gaussians. It is clearly integrable and remarkably simple. The mixing weights ω follow a Dirichlet, the vector of standard deviations σ has independent components that follow inverse-Gamma distributions, and the vector of means μ, conditionally on σ, also has independent components that follow gaussian distributions. Independent samples from this distribution can be obtained exactly, without the need of asymptotic Gibbs sampling, with very efficient procedures in the public domain.

Posterior

Model (82) becomes very useful when assuming that the actual observed data is D = x^{n} = (x_{1},...,x_{n}) such that (x_{1},z_{1}),...,(x_{n},z_{n}) are independent samples from a distribution in (82) but the vector of labels z^{n} = (z_{1},...,z_{n}) is not available, it is missing. The accurate identification of the vector of missing labels is often of great practical importance with applications scattered all over the spectrum of modern science and technology. Estimation of the missing labels, as always, is done by simply wiggling the bayesian wand, i.e., applying bayes theorem. Bayes theorem gives the chances of what we don’t know given what we do know. There is a caveat: A model (i.e. a hypothesis space) and a prior on the model are needed. In the case of mixtures of univariate gaussians we now have all the ingredients.

The Complete Posterior

If we knew the vector of labels z^{n} then the unnormalized complete posterior would be,

π(θ|x^{n},z^{n},α,θ̂) = π(θ|α,θ̂) ∏_{i=1}^{n} \frac{ω_{z_{i}}}{σ_{z_{i}}} exp\{ −\frac{(x_{i}− μ_{z_{i}})^{2}}{2σ_{z_{i}}^{2}} \}  (107)

Let k_{j} be the total number of labels that are equal to j, i.e.,

k_{j} = ∑_{i=1}^{n} 1(z_{i} = j)  (108)

Grouping common labels together we can simplify (107) as,

π(θ|x^{n},z^{n},α,θ̂) = π(θ|α,θ̂) ∏_{j=1}^{k} \frac{ω_{j}^{k_{j}}}{σ_{j}^{k_{j}}} exp\{ −\frac{1}{2σ_{j}^{2}} ∑_{z_{i}=j} (x_{i}− μ_{j})^{2} \}  (109)

Bringing in the prior (106) we obtain the complete posterior as,

π(θ|x^{n},z^{n},α,θ̂) = ∏_{j=1}^{k}  exp\{ −\frac{1}{2σ_{j}^{2}} [αw_{j}(μ_{j} − m_{j})^{2}+ ∑_{z_{i}=j} (x_{i}− μ_{j})^{2}]\}

·σ_{j}^{−αw_{j}−2−k_{j}} exp\{ −\frac{αw_{j}s_{j}^{2}}{2σ_{j}^{2}} \}  (110)

·ω_{j}^{αw_{j}+k_{j}+ \frac{1}{2}}

Again, independent samples of θ with distribution (110) can be obtained exactly, without the need of Gibbs iterations. We factorize the complete posterior distribution of θ as,

(θ) = (μ|σ) (σ) (ω)  (111)

Thus, to sample a vector θ = (μ,σ,ω) from (110) we first produce ω with,

ω ∼ D(αw_{1}+ k_{1}+ \frac{3}{2},αw_{2}+ k_{2}+ \frac{3}{2},...,αw_{k}+ k_{k}+ \frac{3}{2})  (112)

where D stands for the Dirichlet distribution. Then we sample σ from its unconditional distribution obtained from (110) by integrating over μ. We collect the resulting distribution in the following theorem,

Theorem 16 In the complete posterior (110) the distribution of the vector σ is,

π(σ|x^{n},z^{n},α,θ̂) = ∏_{j=1}^{k} σ_{j}^{−αw_{j}−k_{j}−1} exp\{ \frac{−1}{2σ_{j}^{2}} [αw_{j}s_{j}^{2}+ A_{j}]\}  (113)

where,

A_{j} = k_{j} [V_{j}^{2}+ \frac{αw_{j}}{αw_{j} + k_{j}} (x̄_{j} − m_{j})^{2}]  (114)

and

V_{j}^{2} = \frac{1}{k_{j}} ∑_{z_{i}=j} (x_{i}− x̄_{j})^{2} = x̄_{j}^{2}− (x̄_{j})^{2}  (115)

and we are using the notation,

x̄_{j} = \frac{1}{k_{j}} ∑_{z_{i}=j} x_{i}  (116)

Proof Tedious but straight forward. We have,

π(σ|x^{n},z^{n},α,θ̂) = ∫ π(θ|x^{n},z^{n},α,θ̂) dμ

Looking at (110) we see that we need to evaluate the integrals,

J_{j} = ∫ exp\{ −\frac{1}{2σ_{j}^{2}} [αw_{j}(μ_{j} − m_{j})^{2}+ ∑_{z_{i}=j} (x_{i}− μ_{j})^{2}]\} dμ_{j}

Expanding the squares, collecting terms involving μ_{j}, completing back the square, performing the gaussian integral, and simplifyiiing we obtain,

J_{j} = \frac{σ_{j}}{\sqrt{αw_{j} + k_{j}}} exp\{ \frac{−1}{2σ_{j}^{2}} A_{j}\}  (117)

Q.E.D. Also,

Theorem 17 With the notation of theorem (16). For j = 1,2,...,k let,

σ_{j} = \sqrt{\frac{αw_{j}s_{j}^{2}+ A_{j}}{2τ_{j}}}  (118)

If τ_{1},τ_{2},...,τ_{k} are chosen independently with distributions,

τ_{j} ∼ G (\frac{αw_{j} + k_{j}}{2} )  (119)

where G denotes a Gamma distribution. Then the vector σ = (σ_{1},...,σ_{k}) follows the distribution (113).

Proof Just change the variables. Q.E.D.

Finally, once the vector σ is given, we sample μ from the conditional distribution,

π(μ|σ,x^{n},z^{n},α,θ̂) = ∏_{j=1}^{k} N(μ_{j};μ̃_{j} ,σ̃_{j} )  (120)

where,

μ̃_{j} = \frac{αw_{j}m_{j} + k_{j} x̄_{j}}{αw_{j} + k_{j}} and σ̃_{j} = \frac{σ_{j}^{2}}{αw_{j} + k_{j}}  (121)

This is the standard computation of the posterior distribution of a gaussian mean when the likelihood and the prior are both gaussian.

Gibbs Sampler

To sample the actual posterior π(θ|x^{n},α,θ̂) we need to sample from the joint distribution of (θ,z^{n}) and discard the labels z^{n}. We use a Gibbs sampler to eventually sample vectors, (θ,z^{n}) by iterating samples from the two conditional distributions: The complete posterior π(θ|x^{n},z^{n},α,θ̂), as indicated above, and the posterior distribution for the labels, given by

p(z^{n}|θ,x^{n},α,θ̂) ∝ p(x^{n},z^{n}|θ,α,θ̂) = ∏_{i=1}^{n} p(x_{i},z_{i}|θ)

∝ ∏_{i=1}^{n} \frac{ω_{z_{i}}}{σ_{z_{i}}} exp\{ −\frac{(x_{i}− μ_{z_{i}})^{2}}{2σ_{z_{i}}} \}  (122)

Equation (122) shows that conditionally on (θ,x^{n}), the labels z_{1},z_{2},...,z_{n} are independent with probabilities, for j = 1,2,...,k given by

P[z_{i} = j|θ,x^{n}] = \frac{1}{c} \frac{ω_{j}}{σ_{j}} exp\{ −\frac{(x_{i}− μ_{j})^{2}}{2σ_{j}^{2}} \}  (123)

where c > 0 is the normalizing constant. Notice also, that estimation of the labels can be done by using the same joint samples of (θ,z^{n}) but now discarding θ instead of z^{n}. The observed z^{n} will be samples from p(z^{n}|x^{n},α,θ̂).

MODEL SELECTION

Having been presented with two alternative models M_{0} and M, possibly of different dimensions, for the same vector x^{n} = (x_{1},...,x_{n}) of observations, when should we choose M over M_{0}?. This is a decision problem. Once we choose a loss function L and prior probabilities for the two alternative hypotheses M_{0} and M the optimal decision is the one given by bayes rule: choose the model that minimizes the expected loss conditional on whatever is known.

The optimal values of the invariant actions for ignorance that we have found in this paper provide the decision maker with a natural class of loss functions. For fix values of α,δ and ν we can quantify the loss of deciding to use model M when the true distribution is t by,

L(t,M;α,δ,ν) = A_{M}^{∗} = A_{M}(π^{∗};α,δ,ν,t,O)  (124)

Thus, we have a collection of rankings for models. In the absence of any observations, when the true distribution is t we choose model M over M_{0} when A_{M}^{∗} < A_{M_{0}}^{∗}. We have possibly different rankings for different choices of the parameters α,δ, and ν. But these are only theoretical, ideal rankings, for they depend on the unknown true distribution t. When a vector x^{n} of observations is available it seems reasonable to estimate the unknown t by t̂(x^{n}), e.g. with the M.L.E., in each model. The obvious question is: How good is that?. We show below that the case δ = ν = 0 is asymptotically equivalent to the plain bayesian ranking, i.e. the one that chooses the model with the highest posterior probability.

In fact, given two alternative models M_{0} and M, observed data x^{n}, prior probabilities P(M_{0}) and P(M) and prior distributions on the models, π_{M} and π_{M_{0}} the plain bayesian ranking chooses M over M_{0} iff

P(M|x^{n}) > P(M_{0}|x^{n})  (125)

By bayes theorem the posterior probabilities are given by,

P(M|x^{n}) = \frac{1}{Z}p(x^{n}|M)P(M)  (126)

where, Z is a normalizing constant independent of the model M. The marginal likelihood can be expanded as,

p(x^{n}|M) = ∫ p(x^{n},θ|M) dV (θ)
= ∫ p(θ|M)p(x^{n}|θ,M) dV (θ)
= ∫_{M} π_{M}(p)∏_{i=1}^{n} p(x_{i}) dV (p)  (127)
= ∫_{M} π_{M}(p)e^{nL_{n}(p)} dV  (128)

and by the standard central limit theorem,

L_{n}(p) = \frac{1}{n} ∑_{i=1}^{n} logp(x_{i})
= ∫ t(x)logp(x) dx + \frac{σ_{t}}{\sqrt{n}}N(0,1) + o(n^{−1/2} )
= −I_{0}(p: t) + ∫ t(x)logt(x) dx + O(n^{−1/2} )  (129)

Thus, when P(M_{0}) = P(M) replacing (129) into (128) we obtain that the plain bayesian ranking will choose among the alternative models the M that minimizes,

−logP(M|x^{n}) = −log∫_{M} π_{M}(p)e^{−n(I_{0}(p:t)+O(n^{−1/2} ))} dV + (constant)  (130)

This shows that for large n, neglecting lower order terms and the irrelevant additive constant, the model M with the highest posterior probability is the one that minimizes,

L_{M} = −log∫_{M} π_{M}(p)e^{−nI_{0}(p:t)} dV  (131)

When M has finite volume V < ∞ the most ignorant prior on M is the uniform (relative to dV) given by π_{M}(p) = 1/V and we have,

L_{M} = A_{M}^{∗}|_{α=n,δ=ν=0}  (132)

When M has infinite volume, we replace Jeffreys prior with an ignorant prior π_{M} = [1 + ανI_{δ}]^{−1/ν} with α → α_{min}. The resulting objective function L_{M} can still be thought as an A_{M}^{∗} if we replace in (14) the uniform ω with its approximation π_{M}.

Ignorant Razors

The previous discussion suggests interpreting the optimal values of the actions for ignorance A_{M}^{∗} as measures of model accuracy and complexity. A general family of Occam’s razors is given by,

R(M) = \frac{∫_{M}e^{−ᾱIδ̄(p:t)}[1 + ανI_{δ}(p: t)]^{−1/ν} dV}{∫_{M}[1 + ανI_{δ}(p: t)]^{−1/ν} dV}  (133)

The case ν = 0 has the simplest form,

R(M) = \frac{∫_{M}e^{−(α+β)I_{δ}(p:t)} dV}{∫_{M}e^{−αI_{δ}(p:t)} dV}  (134)

obtained when ᾱ = β and δ̄= δ. In this case,

A_{M}^{∗} = −log \frac{Z_{M}(α + β)}{Z_{M}(α)}  (135)

where,

Z_{M}(α) = ∫_{M}e^{−αI_{δ}(p:t)} dV  (136)

By imitating Balasubramanian’s paper we produce a perturbative expansion of the normalizing constants of type (136) as α → ∞. We should expect to see the effects of the δ-geometry explicitly in this expansion since the δ-connection coefficients and its derivatives are obtainable from the Eguchi relations introduced in the IG overview.

With the help of a parameterization we write,

Z_{M}(α) = ∫ e^{−αE(θ)} dθ  (137)

where,

E(θ) = I_{δ}(θ : t) − α^{−1} log\sqrt{detI(θ)}  (138)
= J(θ) − α^{−1} F (θ)  (139)

Let us now define,

θ^{δ}= argmin_{θ}I_{δ}(θ : t)  (140)

Now recall that α is interpreted as a continuous number of apriori independent observations from the true distribution t (see (79)). Hence, the central limit theorem provides a justification for an α-dependent scaling of the variable of integration. Let r = α^{−1/2} and define the new variable of integration,

ϕ = \sqrt{α} (θ− θ^{δ}) ⟺ θ =θ^{δ} +rϕ  (141)

so that,

Z_{M}(α) = α^{−κ/2} ∫ e^{−αE(θ^{δ}+rϕ)} dϕ  (142)

where κ is the dimension of the manifold model M. Considering the exponent as a function of r > 0 and expanding the functions J and F in a Taylor series about r = 0 we get,

E(θ^{δ} +rϕ) = E(θ^{δ}) + ∑_{j=2}^{∞} \frac{r^{j}}{j!} J_{μ_{1}··· μ_{j}} ϕ^{μ_{1}} ··· ϕ^{μ_{j}} − r^{2} ∑_{i=1}^{∞} \frac{r^{i}}{i!}F_{μ_{1}··· μ_{i}} ϕ^{μ_{1}} ··· ϕ^{μ_{i}}

= E(θ^{δ}) + \frac{α^{−1}}{2!} J_{μ_{1}μ_{2}} ϕ^{μ_{1}}ϕ^{μ_{2}} + α^{−1} G(ϕ)  (143)

where,

J_{μ_{1}··· μ_{j}} = ∂_{μ_{1}} ··· ∂_{μ_{j}} I_{δ}(θ : t)|_{θ=θ^{δ}}  (144)
F_{μ_{1}··· μ_{i}} = ∂_{μ_{1}} ··· ∂_{μ_{i}}F (θ)|_{θ=θ^{δ}}  (145)
G(ϕ) = ∑_{i=1}^{∞} α^{−i/2} [ \frac{1}{(i + 2)!} J_{μ_{1}··· μ_{i+2}} ϕ^{μ_{1}} ··· ϕ^{μ_{i+2}} − \frac{1}{i!}F_{μ_{1}··· μ_{i}} ϕ^{μ_{1}} ··· ϕ^{μ_{i}}]  (146)

and we have used the fact that when j = 1 the rhs of (144) is zero. It is easy to verify that equations (144) and (145) are directly connected to the δ-geometry. In fact, we have

J_{ij}|_{t=θ^{δ}} = g_{ij}(θ^{δ})  (147)
J_{ijk}|_{t=θ^{δ}} = Γ^{δ}_{ijk} (θ^{δ}) + Γ_{jki}(θ^{δ}) + Γ_{kij}(θ^{δ})  (148)
F_{j} = Γ_{ji}^{i}(θ^{δ})  (149)
F_{jk} = Γ_{ji,k}^{i}(θ^{δ}) = Γ_{ki,j}^{i}(θ^{δ})  (150)

Equations (147) and (148) are an alternative form of the Eguchi relations that can be checked by straight forward computation of the derivatives. Connection coefficients appearing without a δ on top are the metric (Levi-Civita) δ = 1/2 case, and partial derivatives in (150) are indicated by the sub index following a comma. Obviously (150) follows from (149) but (149) is not obvious. It is however well known and often left as an exercise in general relativity text books. For the sake of completeness we provide a proof in the following theorem.

Theorem 18 For any metric g_{ij} with associated connection coefficients Γ_{ij}^{k} we have,

∂_{k}log\sqrt{det(g_{ij})} = Γ_{kl}^{l}  (151)

Proof The metric G = (g_{ij}) is symmetric and positive definite. Thus, there exists an orthogonal matrix P, i.e. P^{T}P = Id, with G = P^{T}DP and D a diagonal matrix with positive entries λ_{i}. Hence,

∂_{k}log\sqrt{det(g_{ij})} = \frac{1}{2}∂_{k}log detG = \frac{1}{2}∂_{k}( ∑_{i} logλ_{i})

= \frac{1}{2} ∑_{i} \frac{1}{λ_{i}} ∂_{k}λ_{i} = \frac{1}{2}tr D^{−1} ∂_{k}D
= \frac{1}{2}tr (P^{T}D^{−1} PP^{T}∂_{k}DP) = \frac{1}{2}tr G^{−1} ∂_{k}G

= \frac{1}{2}g^{im}g_{mi,k}
= g^{im}\frac{1}{2}(g_{im,k}+ g_{mk,i}− g_{ki,m}) + \frac{1}{2} \{g^{im}(g_{ki,m}− g_{mk,i})\}
= Γ_{ki}^{i}  (152)

where we have used the cyclic property of the trace, the definition of the connection coefficients Γ_{ijk} = \frac{1}{2}(g_{jk,i} + g_{ki,j} −g_{ij,k}) and the fact that the expression within curly brackets is zero. This last fact follows from differentiating the constant identity ∂_{j}(g^{im}g_{mk}) = 0 so that,

g^{im}(g_{ki,m}− g_{mk,i}) = (−g_{,l_{1}}^{l_{1}l_{2}}g_{kl_{2}}) − (−g_{,l_{1}}^{l_{1}l_{2}}g_{kl_{2}}) = 0

Q.E.D. Now from the fact that,

∂_{h_{μ_{1}}} ··· ∂_{h_{μ_{k}}}e^{h_{i}ϕ^{i}} = e^{h_{i}ϕ^{i}}ϕ^{μ_{1}} ··· ϕ^{μ_{k}}  (153)

it follows, by writing the exponential as a power series, that

e^{−G(ϕ)} = e^{−G(∇_{h})} e^{h_{i}ϕ^{i}}|_{h=0}  (154)

where ∇_{h} ≡ (∂_{h_{1}},...,∂_{h_{κ}}), and G(ϕ) is any multivariate series like (146). Thus, replacing (143) into (142) using (154) and moving the derivatives outside the integral, we obtain,^{4}

Z_{M}(α) = α^{−κ/2} e^{−αI_{δ}(θ^{δ}:t)} \sqrt{detI(θ^{δ})} e^{−G(∇_{h})} ∫ e^{− \frac{1}{2}J_{ij}ϕ^{i}ϕ^{j}+h_{k}ϕ^{k}} dϕ|_{h=0}  (155)

= α^{−κ/2} e^{−αI_{δ}(θ^{δ}:t)} \sqrt{detI(θ^{δ})}e^{−G(∇_{h})} [( \frac{(2π)^{κ}}{detJ} ) ^{1/2}exp(\frac{1}{2}h_{i}J^{ij}h_{j})]|_{h=0}  (156)

where the gaussian integral (155) was computed by noticing that if we let m^{i} = h_{k}J^{ki} then we can write the integrand as,

exp\{ −\frac{1}{2}J_{ij}ϕ^{i}ϕ^{j} + h_{k}ϕ^{k}\} = exp\{ \frac{1}{2}J_{ij}m^{i}m^{j}\}exp\{ −\frac{1}{2}(ϕ^{i}− m^{i})J_{ij}(ϕ^{j} − m^{j})\}
= exp\{ \frac{1}{2}J^{ij}h_{i}h_{j}\}exp\{ −\frac{1}{2}(ϕ^{i}− m^{i})J_{ij}(ϕ^{j} − m^{j})\}  (157)

Let us re-write (146) as,

G(ϕ) = −∑_{i=1}^{∞} α^{−i/2} A_{i}(ϕ)  (158)

Using (158) and (156) we obtain an expansion of Z_{M}(α) to all orders. For example, if we neglect terms of order α^{−2} and smaller we have,

e^{−G(ϕ)} = 1 − G(ϕ) + \frac{G^{2}(ϕ)}{2!} − \frac{G^{3}(ϕ)}{3!} + ···
= 1 + [α^{−1/2} A_{1}(ϕ) + α^{−1} A_{2}(ϕ) + α^{−3/2} A_{3}(ϕ)] +

4 This is a very powerful trick that I learned from Balasubramanian’s paper. Notice however that equation (41) in that paper is missing a factor −1/2 in the exponent.

\frac{1}{2} \{α^{−1} A_{1}^{2}(ϕ) + 2α^{−3/2} A_{1}(ϕ)A_{2}(ϕ)\} +

\frac{1}{6}α^{−3/2} A_{1}^{3}(ϕ) + O(α^{−2} )  (159)

the terms within square brackets correspond to the terms in −G(ϕ) of order lower than α^{−2} , those within curly brackets are from G^{2}(ϕ), and the last term is, the only other contribution of order lower than α^{−2} , from G^{3}(ϕ)/3!. It can be shown that the terms with an odd number of phi’s, like the terms of orders α^{−1/2} and α^{−3/2} will end up not contributing anything when (159) is transformed into a differential operator and used in (156). Thus, for model selection, the choice of δ-geometry will appear in the terms of order α^{−1} . Let us use the following convenient notation,

f = exp( \frac{1}{2}J^{μν} h_{μ}h_{ν})  (160)
m^{j} = J^{μj} h_{μ}  (161)

Thus, using (159) we obtain,

e^{−G(∇_{h})} f|_{h=0} = 1 + α^{−1} \{ A_{2}(∇_{h}) + \frac{1}{2}A_{1}^{2}(∇_{h})\} f| _{h=0} + O(α^{−2} )
= 1 + α^{−1} \{ [\frac{1}{2}Γ_{iμ,j}^{μ} T^{ij} − \frac{1}{4!}J_{ijkl}T^{ijkl}]+

\frac{1}{2}[Γ_{iμ}^{μ}Γ_{jν}^{ν} T^{ij} − \frac{2}{3!}Γ_{iμ}^{μ}J_{jkl}T^{ijkl} + ( \frac{1}{3!}) ^{2}J_{ijk}J_{lmn}T^{ijklmn}]\}+ O(α^{−2} )  (162)

where the first term within square brackets is from A_{2} and the second is from A_{1}^{2} and,

T^{μ_{1}...μ_{i}} = ∂_{h_{1}}··· ∂_{h_{i}}f|_{h=0}  (163)

Applying the formulas ∂_{j}f = fm^{j} and ∂_{i}m^{j} = J^{ij} recursively we obtain, after a long but straight forward computation, that

T^{ij} = J^{ij}  (164)
T^{ijkl} = J^{ij}J^{kl}+ J^{ik}J^{jl} + J^{il}J^{jk}  (165)
T^{ijklmn} = J^{ij}J^{kl}J^{mn}+ (..11 similar terms)  (166)

Substituting into (162) and simplifying we find,

e^{−G(∇_{h})} f|_{h=0} = 1 + \frac{α^{−1}}{24} J^{ij}S_{ij} + O(α^{−2} )  (167)

where,

S_{ij} = 12Γ_{iμ,j}^{μ} − 3J_{ijkl}J^{kl}− 3Γ_{iμ}^{μ}Γ_{jν}^{ν} − 12Γ_{iμ}^{μ}J_{jkl}J^{kl}+ 4J_{ijk}J_{lmn}J^{kl}J^{mn}  (168)

The above tensor is closely related to the Ricci tensor. As it can be readily checked (see MTW p.222, (8.51a,b)), the Ricci tensor can be written as,

R_{ij} = −Γ_{iμ,j}^{μ} + Γ_{ij,μ}^{μ} + Γ_{μν}^{ν} Γ_{ij}^{μ} − Γ_{νi}^{μ}Γ_{jμ}^{ν}  (169)

We can therefore write (168) in terms of the Ricci tensor as,

S_{ij} = −12(R_{ij} + T_{ij})  (170)

where the new tensor T_{ij} collects all the left-over terms. We can regard the above equation as the defining equation for T_{ij}. We expect T_{ij} to have zero trace when the model contains the true distribution. More detailed computations will be the subject of an upcoming paper. With this notation we can write once again (167) in the following more useful form,

e^{−G(∇_{h})} f|_{h=0} = 1 − \frac{α^{−1}}{2} J^{ij}[R_{ij} + T_{ij}] + O(α^{−2} )  (171)

From (171), (159) and (156) we get,

−logZ_{M}(α) = α I_{δ}(θ^{δ}: t)  (172)
+\frac{κ}{2} log \frac{α}{2π}  (173)

+\frac{1}{2} log( \frac{detJ(θ^{δ})}{detI(θ^{δ})})  (174)

−log( 1 − \frac{α^{−1}}{2} J^{ij}[R_{ij} + T_{ij}] + O(α^{−2} ))  (175)

This formula provides a new measure of complexity for models whose volumes have been normalized. If the models to be compared occupy different volumes in the space of distributions then we need to add logV_{M} as a fifth term to the above formula. The first three terms for the case δ = 0 have been previously obtained in connection to the so called Minimum Description Length (MDL) principle, attributed to Rissanen. The general δ case, and specially the fourth term involving the scalar built out of the curvature connection coefficients, were previously unknown.

Each of the four terms of the above expansion has a clear interpretation. The leading term (172) measures the accuracy of the model M in terms of the δ-distance from the model to the true distribution and increases linearly with the amount of prior information (data) α. This first term ensures asymptotic correctness (consistency) as α → ∞, i.e. as more and more prior information is available. The second term (173) penalizes models of high dimension and encodes parsimony relative to the amount of available prior information. It is linear in the dimension κ but logarithmic in α. This term was first discovered by Schwartz in connection with the so called Bayesian Information Criterion (BIC). This term does not depend on δ in any way. The third term, constant in α, encodes what Balasubramanian calls the naturalness or robustness of the model. It is a penalty on models that have few indistinguishable points that are close to the true one, e.g. when the model looks highly curved from the point of view of the true distribution. The fourth term (175), let us call it S_{4}, is the most remarkable for it has not previously appeared in the literature. Expanding the logarithm and neglecting the lower order terms we have,

S_{4} = \frac{α^{−1}}{2} J^{ij} [R_{ij} + T_{ij}]  (176)

We remind the reader that the above quantities are evaluated at a special point in M, namely the closest point to t in the δ-deviation distance. Also, J depends on the true distribution t as well.

When M ∋ t

When the model M is sufficiently large or informative, so that we can assume that it contains, or at least closely approximates, the true distribution the action for model selection takes a simpler form.

In fact when t =θ^{δ} we have from (147) that J^{ij} = g^{ij} and the terms (172) and (174) become zero. Hence, as α → ∞

−logZ_{M}(α) = \frac{κ}{2} log \frac{α}{2π} − log( 1 − \frac{α^{−1}}{2} g^{ij}[R_{ij} + T_{ij}]|_{θ=θ^{δ}} ) + O(α^{−2} )  (177)

Now consider π to be any prior scalar density that peaks at θ^{δ} as α → ∞. For example a natural choice is given by the ignorant (δ,0)-prior,

π(θ) = e^{−α I_{δ}(θ:θ^{δ})}  (178)

For any such prior π the action for model selection (177) can be thought as an approximation to the global action given mod α^{−2} by:

−logZ_{M}(α) = \frac{κ}{2} log( \frac{α}{2π} ) − log( 1 − \frac{α^{−1}}{2} ∫_{M}(R + T)π dV )  (179)

where R is the Ricci scalar and T is the trace of T_{ij} plus any term, which may depend on π, that becomes negligible around the peak. Notice that when comparing models with the same number of parameters κ, the first term in (179) becomes an irrelevant constant and the action becomes equivalent to the integral, i.e., the prior expectation of (R+T).

The practical utility of the new ranking procedures introduced in (179), (177) or the original (172-175) need to be tested on real and simulated data. Preliminary computer experiments of the effect of the geometry for model selection, just at the level of correct volume normalization in the MDL principle, are currently being investigated by Eitel Lauria and myself. These experiments show that adding (to the MDL score) a simple approximation to the logarithm of the total volume occupied by a binary bayesian network of a given structure never hurts. In fact the modified score chooses the correct model in (on the average) about 30% more cases than the standard MDL score across a wide range of structures and sample sizes. These results are encouraging but nothing is known, at the moment of writing, about the actual performance of the new curvature term introduced in this paper.

AUTOMATIC MODEL PRODUCTION

In this section we show how the actions for model selection, introduced in the previous section, can be transformed into actions for the automatic production of statistical models. The resulting theory seems to be completely unexplored and extremely rich. What follows is just a very first attempt at describing what seems to me as an ocean of new statistical questions.

Simply put. Take the exp of (179) and elevate the obtained score to the status of universal ranking for all the unknowns, i.e., the dimension κ, the information metric g_{ij} and the unnormalized prior π, or equivalently its log likelihood f, where π = e^{−f} . When T = |∇f|^{2} and τ = (2α)^{−1} the obtained score is equivalent mod α^{−2} to minimizing,

F = ∫_{M}[τ(R + |∇f|^{2}) − \frac{κ}{2}] (4πτ)^{−κ/2} e^{−f} dV  (180)

over smooth functions f satisfying the normalization condition,

∫_{M}(4πτ)^{−κ/2} e^{−f} dV = 1  (181)

Perelman’s Action

In a recent paper^{5} Grisha Perelman shocked the topology world with what the experts say may be a final proof of the so called geometrization conjecture for three-manifolds. This implies the famous Poincaré conjecture. It is much more famous now that there is a $1M prize for its proof^{6}. What shocks this pedestrian statistician is that (180) is the action that produces the flow of metrics g_{ij}(τ) that is able to smooth out all manifolds and prove that topologically, spheres are not doughnuts in any dimensions. I believe that the entropy formula for Ricci flows, that solves the presumably esoteric ball ≠ doughnut problem has an statistical interpretation with the potential for unifying not just heaven and earth but the mind with the moon. In the mean τime, Truth, time & τemperature

5 see arXiv:math.DG/0211159 6 see www.claymath.org

time is cold Truth

and

Truth is hot time.

Red hot is a true first second

of love.

But all Truth is temporal.

Ergo, F$$$. that’s true

capitalism. T is t and τ τ & t is T & T T is t and τ is T god save TT&T god save τT&t god save Tτ&t god save the t.

ACKNOWLEDGMENTS

REFERENCES

1. Brown, M. P., and Austin, K., The New Physique, Publisher Name, Publisher City, 2000, pp. 212–213.

2. Brown, M. P., and Austin, K., Appl. Phys. Letters, 85, 2503–2504 (2000).

3. Mittelbach, F., and Schöpf, R., TUGboat, 11, 297–305 (1990), URL http://www.latex-project.org.

4. Wang, R., “Title of Chapter,” in Classic Physiques, edited by R. B. Hamil, Publisher Name, Publisher City, 2000, pp. 212–213.

5. van Herwijnen, E., Future Office Systems Requirements, Tech. rep., CERN DD Internal Note (1988).

6. Liang, F. M., Word Hy-phen-a-tion by Com-pu-ter, Ph.D. thesis, Stanford University, Stanford, CA 94305 (1983), also available as Stanford University, Department of Computer Science Report No. STAN-CS-83-977.

7. Smith, C. D., and Jones, E. F., “Load-Cycling in Cubic Press,” in Shock Compression of Condensed Matter-1999, edited by M. D. F. et al., AIP Conference Proceedings 505, American Institute of Physics, New York, 1999, pp. 651–654.

8. Knuth, D. E., The WEB System of Structured Documentation, Tech. Rep. STAN-CS-83-980, Department of Computer Science, Stanford University, Stanford, CA 94305 (1983).

A Geometric Theory of Ignorance  July 31, 2003  30