11. Posterior Using Gibbs' Sampler

Approx. Using Gibbs' Sampler

Example

Parameters: $σ^{2}$ and $μ$
Likelihood: $P (y_{i} | μ, σ^{2})$ is $N (μ, σ^{2})$
Prior: $p (μ, σ^{2}) \overset{ind.}{=} p (μ) p (σ^{2})$

$p (μ) = 1$ and $p (σ^{2}) = \frac{1}{σ^{2}} = (σ^{2})^{- 1}$ Jeffreys' Prior

p ({y}) = {[\frac{1}{2 π}]}^{n / 2} (σ^{2})^{- n / 2} \exp {- [\frac{\sum_{}^{} (y - μ)^{2}}{2}] [\frac{1}{σ^{2}}]}

\begin{array}{r} ⟹ p (μ, σ^{2} | {y}) \propto (σ^{2})^{- n / 2 - 1} \exp {- [\frac{\sum_{}^{} (y - μ)^{2}}{2}] [\frac{1}{σ^{2}}]} \end{array}

Key idea: we have a bivariate probability distribution.

Want to find the "full" conditional for $σ^{2}$ :

\begin{array}{r} p (σ^{2} | {y}, μ) \sim inv. gamma (r_{*} = \frac{n}{2}, v_{*} = \frac{\sum_{}^{} (y - μ)^{2}}{2}) \end{array}

"Full" conditional for $μ$ :

\begin{aligned} p (μ | {y}, σ^{2}) & \propto \exp {- [\frac{\sum_{}^{} (y - μ)^{2}}{2}] [\frac{1}{σ^{2}}]} \\ \propto \exp {\frac{\sum_{}^{} ((y - \bar{y}) + (\bar{y} - μ))^{2}}{- 2 σ}} \\ \propto \exp {\frac{\sum_{}^{} (y - \bar{y})^{2} + 2 \sum_{}^{} (y - \bar{y}) (\bar{y} - μ) + \sum_{}^{} (\bar{y} - μ)^{2}}{- 2 σ}} \\ \propto \exp {\frac{\sum_{}^{} (y - \bar{y})^{2} + n (\bar{y} - μ)^{2}}{- 2 σ}} \\ \propto \exp {- \frac{1}{2 (\frac{σ^{2}}{n})} [μ - \bar{y}]^{2}} \end{aligned}

$N (mean = \bar{y}, variance = \frac{σ^{2}}{n})$

General Idea:
Each step in algorithm sample from each univariate distribution for each parameter and simultaneously update.

\begin{aligned} 1. & sample ϕ_{1}^{(s)} \sim p (ϕ_{1} ∣ ϕ_{2}^{(s - 1)}, ϕ_{3}^{(s - 1)}, \dots, ϕ_{p}^{(s - 1)}) \\ 2. & sample ϕ_{2}^{(s)} \sim p (ϕ_{2} ∣ ϕ_{1}^{(s)}, ϕ_{3}^{(s - 1)}, \dots, ϕ_{p}^{(s - 1)}) \\ ⋮ \\ p . & sample ϕ_{p}^{(s)} \sim p (ϕ_{p} ∣ ϕ_{1}^{(s)}, ϕ_{2}^{(s)}, \dots, ϕ_{p - 1}^{(s)}) \end{aligned}

Pr (ϕ^{(s)} \in A) \to \int_{A} p (ϕ) d ϕ as s \to \infty .