Basic Bayesian Theory

Foreword¶

We will continue where we left off. In the last post, I introduced a few important observables along with the setup. We are interested in their asymptotics, and the behaviors of these random variables. But before we dive into them, we first see some motivation for their definitions, and how they are all connected to each other (the WAIC definition was pretty non-intuitive to me the first time I saw it). Somewhere along the middle, I will slightly detach from Watanabe’s notation to do this and approach this topic differently, before converging back to his notation. This is primarily done to make the relationships between the observables clearer, and to give a more detailed introduction. Once this is done, we will start with the asymptotics.

How do we see the relationships between those observables though, and even try to begin their asymptotics? To help you get settled with the mindset, I will give you the spoiler beforehand: Generating Functions.

For an arbitrary triple of a true distribution, a statistical model, and a prior, the behaviors of the free energy, the losses, and WAIC are derived by the following procedure:

1. Firstly, we define the formal relation between a true distribution and a statistical model.

2. Secondly, definitions of Bayesian observables and their normalized versions are introduced.

3. Thirdly, the cumulant generating functions of the Bayesian prediction are defined.

4. The Basic Theory of Bayesian Statistics is proved using the cumulant generating functions.

The Formal Relation¶

We make 4 main definitions right now.

Definition 1 (Realizability): Let $W \subset \mathbb{R}^d$ be the set of all parameters. If there exists $w_0 \in W$ such that $q(x) = p(x|w_0)$ almost surely, then $q(x)$ is said to be realizable by $p(x|w)$. For a given pair of true distribution and statistical model, the set of true parameters is defined by

$$W_{00} = \{w \in W: q(x) = p(x|w) \text{ for arbitrary } x \text{ s.t. } q(x) > 0\}$$

(1)

Thus, realizability holds iff the set of true parameters is not an empty set (one can see this by using the fact that the integral of both the functions is 1 and then using the almost sure condition).

We can alternatively write

The average log loss function is defined by

It follows that

where $S = -\int q(x)\log q(x)dx$ is the entropy of the true distribution and $K(\cdot\,\|\,\cdot)$ denotes the KL divergence. Hence, if $q(x)$ is realizable by a statistical model, then the average log loss function is minimized iff $w \in W_{00}$, and the minimum value is the entropy of the true distribution.

Some motivation: As we continue the learning process, our statistical model gets closer to the true distribution. The realizability assumption makes it simple for us and tells us that our statistical model can represent the true distribution, which need not be the case in general.

Definition 2 (Regularity): For a given pair of $q(x)$ and $p(x|w)$, let $W_0 = \{w \in W: L(w) = \min_{w'}L(w')\}$, which is called the set of optimal parameters for the minimum average log loss. $q(x)$ is said to be regular for $p(x|w)$ if the following three conditions hold:

1. $W_0$ is a singleton set.

2. $w_0 \in W_0$ is in the interior of $W$ ($W$ is equipped with the subspace topology from $\mathbb{R}^d$).

3. The Hessian matrix of the average log loss function at $w_0$ is positive definite.

Let us see a simple consequence of these definitions. If $W$ is a compact set and if $L(w)$ is a continuous function, then $L(w)$ has a minimum point, hence $W_0$ is not an empty set. In this case, $q(x)$ is realizable by $p(x|w)$ iff $W_{00} = W_0$ (reason this by yourself).

Some motivation: The regularity condition tells us that we can approximate the loss landscape near its minimum by a parabola. This follows by doing a Taylor approximation, doing an eigendecomposition of the Hessian, and then using the fact that all the eigenvalues are positive (do the exact calculations yourself). One of the main claims is that this is a terrible approximation in many cases and does not hold for the loss landscapes of general neural networks.

Definition 3 (Essential Uniqueness): Assume that $W_0$ is not an empty set. If there exists a unique probability density function $p_0(x)$ such that for arbitrary $w_0 \in W_0$, $p(x|w_0) = p_0(x)$, then it is said that the optimal probability density function is essentially unique.

If $q(x)$ is realizable by $p(x|w)$, then the optimal probability density function is essentially unique, because $p_0(x) = q(x)$. Thus, realizability $\implies$ essential uniqueness.

Definition 4 (Relatively Finite Variance of Log Density Ratio Function): For a given pair $w_0 \in W_0$ and $w \in W$, the log density ratio function is defined by

$$> f(x, w_0, w) = \log \dfrac{p(x|w_0)}{p(x|w)} >$$

(5)

If there exists $c_0 > 0$ such that for an arbitrary pair $w_0, w$,

$$> \mathbb{E}_X[f(x, w_0, w)] \geq c_0 \mathbb{E}_X[f(x, w_0, w)^2] >$$

(6)

then the function $f(x, w_0, w)$ has a relatively finite variance.

Remark: The function $f$ has a relatively finite variance iff

Some motivation: What is the purpose of the theory that we are developing? We want to study the model’s distance from the truth, and how that develops over time. We want to abstract out the truth’s randomness from our process. Let us see what this condition is really about.

Let us look at the average log loss function (do you prefer to look at the asymptotics of a sum or a product? Take log.) whose definition is derived naturally from the average loss function.

Removing the entropy considerations from $S$, we get the “excess log loss over the truth per data point”, which is what the definition is in the realizable case (but look at the next lemma to understand the point in the general case). Thus the importance. This remark also says the mathematical statement to note, that the expectation of the log density ratio is the KL divergence between the true distribution and the model $p(x|w)$ in the realizable case. I have explained the ratio, I will explain the condition in a short while.

Lemma: Assume that $w_0 \in W_0$ and $w \in W$. If $f(x, w_0, w)$ has a relatively finite variance, then the optimal probability density is essentially unique.

Assume that $w_1$ and $w_2$ are arbitrary elements of $W_0$. By the definition of $W_0$, $0 = L(w_1) - L(w_2) = \int q(x)f(x, w_1, w_2)dx \geq c_0 \int q(x)f(x, w_1, w_2)^2 dx$ and as the integrand is non-negative, we get $f(x, w_1, w_2) = 0$ almost surely with respect to $q$, hence $p(x|w_1) = p(x|w_2)$ almost surely with respect to $q$. Do observe that this is not a clever proof, rather just making use of all the given information.

Thus, we may denote $f(x, w_0, w) = f(x, w)$. It thus follows that $p(x|w) = p_0(x)\exp(-f(x, w))$.

Thus, relatively finite variance $\implies$ essential uniqueness. However, the converse need not be true. It does hold with a few added conditions.

Lemma: Assume that $W$ is a compact set and that $q(x)$ is realizable by $p(x|w)$ and that the log density ratio function $f(x, w) = \log \dfrac{q(x)}{p(x|w)}$ is a continuous function of $(x, w)$. If there exist $c_1, c_2 > 0$ such that for an arbitrary $w \in W$,

then $f(x, w)$ has a relatively finite variance.

I refer the reader to the book for the proof. It is not important to the rest of the theory, just some technical details.

Lemma: Assume that $W$ is a compact set and that for an arbitrary pair $w_0 \in W_0$ and $w \in W$, the second derivatives of $E_X[f(x, w_0, w)]$ and $E_X[f(x, w_0, w)^2]$ are continuous functions. If $q(x)$ is regular for $p(x|w)$, then the log density ratio function has a relatively finite variance.

The proof of this lemma is alongside the proof of the previous lemma.

Assume that $W$ is compact. We thus have the following relations:

1. $\{$Regular$\} \subset \{$Relatively Finite Variance$\}$.

2. $\{$Realizable$\} \subset \{$Relatively Finite Variance$\}$.

3. $\{$Relatively Finite Variance$\} \subset \{$Essentially Unique$\}$.

┌─────────────────────────────────────────────────────────────┐
│ Essentially Unique                                          │
│                                                             │
│   ┌─────────────────────────────────────────────────────┐   │
│   │ Relatively Finite Variance                          │   │
│   │                                                     │   │
│   │      ┌──────────────────────┐                       │   │
│   │      │ Realizable           │                       │   │
│   │      │                ┌─────┼────────────────┐      │   │
│   │      │                │     │ Regular        │      │   │
│   │      │                │     │                │      │   │
│   │      └────────────────┼─────┘                │      │   │
│   │                       │                      │      │   │
│   │                       └──────────────────────┘      │   │
│   │                                                     │   │
│   └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Normalized Variables¶

We have the average and empirical log loss functions:

Assume now that $f(x, w_0, w) = \log \frac{p(x|w_0)}{p(x|w)}$ has a relatively finite variance. Then the optimal probability density is essentially unique, so we may denote the log density ratio by $f(x, w)$.

See that $L(w) = E_X \left[ \log \frac{1}{p(X|w)} \right] = E_X \left[ \log \frac{p(X|w_0)}{p(X|w)} \right] - E_X [\log p(X|w_0)]$

We define the normalized average log loss function as $E_X[f(X, w)] = K(w)$. A key thing to note in all the definitions is that we will be rewriting them in terms of $f(x, w)$. The formal advantage is explained in a short while, but the understanding is that we want to remove the entropy of the optimal probability density and consider only the KL divergence between $p_0(x)$ and the model $p(x|w)$, as that is what we want to analyse (how that changes over time and what are the asymptotics).

(For this to make sense, note that $E_X [f(X, w)] = K(p_0 \,\|\, p(\cdot|w)) = K(w)$, which equals $K(q \,\|\, p(\cdot|w))$ in the realizable case.)

Similarly, we have the normalized empirical log loss function

Furthermore, a key property of our normalized function is that $K(w) \ge 0$, and $K(w) = 0 \iff w \in W_0$.

The partition function is defined as $Z_n = \int \prod_{i=1}^n p(X_i|w) \varphi(w) dw$

Thus, $Z_n = \int \exp \left( \sum_{i=1}^n \log p(X_i|w) \right) \varphi(w) dw$

Thus, $\frac{Z_n}{\prod_{i=1}^n p_0(X_i)} = \int \exp \left( - \sum_{i=1}^n \log \frac{1}{p(X_i|w)} \right) \cdot \exp \left( - \sum_{i=1}^n \log p_0(X_i) \right) \varphi(w) dw$

We define this as the normalized marginal likelihood $Z_n^{(0)}$.

Thus, $Z_n = \prod_{i=1}^n p_0(X_i) \, Z_n^{(0)} = \exp(-n L_n(w_0)) Z_n^{(0)}$.

Let us recall that the posterior is

Free energy is the negative log of the partition function. The normalized free energy is defined similarly.

Here is the table for the generalization, cross validation, and training losses and the WAIC, and also their normalized versions.

Let us connect all these observables.

Defn: For $\alpha \neq 0$, we define the $\alpha$-predictive at $x$ as

Note that this is the $\alpha$-power mean of $p(x|w)$ with respect to the posterior distribution. Essentially, the observables come from this family.

Recall that the moment generating function of a random variable $X$ with PDF $p(x)$ is given by $M_X(\alpha) = E[e^{\alpha X}]$ and the cumulant generating function is $G_X(\alpha) = \log E[e^{\alpha X}]$

We will look at the cumulant generating function of the log likelihood at $x$ (with a little notational change) wrt the posterior, which is exactly the log of the ($\alpha$-predictive)$^\alpha$.

We can now write the observables in terms of the CGF.

Thus they are all derived from the CGF. We will discuss WAIC shortly, and also note how it comes about.

Let us discuss the normalization first.

We define $F_X(\alpha) = \log E_w [e^{\alpha f(x,w)}]$ in the normalized coordinates.

The corresponding definitions in terms of $F_X(\alpha)$ give the normalized variables.

Note that $F_X(\alpha) = g_x(-\alpha) + \alpha \log p_0(x)$.

The point of introducing the CGF is to be able to do a Taylor expansion here.

Doing a Taylor approximation near 0 (here $g(0) = \log E_w[1] = 0$),

where $g'(0) = E_w [\log p(x|w)] = C_1$ $g''(0) = V_w [\log p(x|w)] = C_2$

So, $g_x(1) + g_x(-1) = C_2 + \frac{g^{(4)}(\theta)}{24} + \frac{g^{(4)}(-\theta)}{24}$

Summing over the training points and writing $R_n$ for the accumulated remainder terms, we get

This gives rise to WAIC, and using the normalized CGF gives rise to the normalized WAIC.

Caveat: Watanabe does an abuse of notation (harmless) and defines it the following way. We will continue with this.

Thus, his definition is just the mean of our definitions. However, do note that the two means are different, since $E_X$ and $E_w$ do not commute.

I will now state the result which explains why we need to look at the normalized observables.

All these asymptotics abstract away from the prior. In the realizable case, they abstract away from the statistical model as well. We need to clarify the behavior of these random variables, look at higher order terms. The normalized versions of these all converge to 0 in probability, removing the effects of $L_n(w_0)$ and $L(w_0)$ and focusing on what is necessary.

We will now explain the asymptotics. We will be considering the normalized observables (and drop the (0) superscript):

Theorem: Let $G'(0), T'(0), G''(0), T''(0)$ be random variables, and assume that

(That is inside radius 1, the third derivatives of the Taylor expansion are bounded).

Then

Proof: By Taylor’s theorem with the Lagrange remainder, given $\alpha$, there exists $\alpha^*$ s.t. $|\alpha^*| \le |\alpha|$ and

$\alpha = 1$ gives the theorem (similar method for dealing with $T_n$).

We thus get the theoretical behaviors of the free energy, generalization loss, and the rest by the following procedure.

Recipe For Bayesian Theory Construction

1. An arbitrary triple of $(q(x), p(x|w), \varphi(w))$ is fixed. The set of parameters is denoted $W$, and $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} q(x)$.

2. The empirical and average log loss functions are defined by

   $$L_n(w) = -\frac{1}{n} \sum_{i=1}^n \log p(X_i|w)$$

   (45)
   $$L(w) = -\int q(x) \log p(x|w) dx$$

   (46)

Find the optimal parameters minimizing $L(w)$.

3. Check that the log density ratio function has a relatively finite variance. Then we have essential uniqueness.

4. Define

   $$K(w) = \int f(x,w) q(x) dx$$

   (48)
   $$K_n(w) = \frac{1}{n} \sum_{i=1}^n f(X_i,w)$$

   (49)

The normalized partition function is given by

Then the free energy $F_n = n L_n(w_0) - \log Z_n^{(0)}$.

5. The average by the posterior is equal to

Calculate $E_w[f(x,w)]$ and $V_w[f(x,w)]$. From these we obtain

and the empirical variance term $\frac{1}{n} \sum_{i=1}^n V_w[f(X_i,w)]$, which estimates $E_X V_w[f(X,w)]$.

Finally, based on the basic theorem,

and $W_n$ has the same expansion as $C_n$ (which by the way proves that $C_n = W_n + O_p(1/n)$).

Further Steps¶

We have already set up the framework and the basic theory, and also discovered important results (which were not restricted to regular models). We are now ready to find the behavior of the regular posterior distribution, and then generalize that behavior to the general case.