4. Bayesian Inference on Poisson

Poisson Distribution

Parameter: $μ$
Likelihood: $p (y | μ)$ is discrete
Poisson:

\begin{array}{r} p (y | μ) = \frac{μ^{y}}{y!} e^{- μ} \\ y = 0, 1, 2, \dots \end{array}

Example 1:

Let Y be the number of accidents occurring in an industrial plant, Y is described by a Poisson process with mean μ accidents every three months. Suppose that 5 possible values of μ are 1/3, 2/3, 1, 4/3, and 5/3. We do not have any reason to give any possible value more weight than any other value, so we give them equal prior weight. During the last three months, NO accidents occur.

a) Find the posterior distribution, i.e., p(μ|data).

\begin{array}{cccc} Possible Values for μ & Prior & Likelihood(Poisson) & Prior \times Likelihood & Posterior \\ \frac{1}{3} & \frac{1}{5} & P (y = 0 | μ = \frac{1}{3}) \approx 0.7165 & \approx 0.1433 & 0.3494 \\ \frac{2}{3} & \frac{1}{5} & P (y = 0 | μ = \frac{2}{3}) \approx 0.54134 & \approx 0.1027 & 0.2504 \\ 1 & \frac{1}{5} & P (y = 0 | μ = 1) \approx 0.3678 & \approx 0.0735 & 0.1795 \\ \frac{4}{3} & \frac{1}{5} & P (y = 0 | μ = \frac{4}{3}) \approx 0.3678 & \approx 0.0527 & 0.1285 \\ \frac{5}{3} & \frac{1}{5} & P (y = 0 | μ = \frac{5}{3}) \approx 0.2636 & \approx 0.0378 & 0.0922 \\ sum= 0.4101 \end{array}

b) Find the posterior mean, i.e., E(μ|data)

E [μ | y = 0] = (\frac{1}{3}) (0.3494) + (\frac{2}{3}) (0.2504) + \dots + (\frac{5}{3}) (0.0922) \approx_{0} .7878

c) Find p(μ ≤ 1|data).

p [μ \leq 1 | y = 0] \approx 0.3494 + 0.2304 + 0.1795 \approx 0.7793

Gamma Prior

A random variable $Y$ is said to have a Gamma distribution with parameters $r > y$ and $v > 0$ $⟺$ the density function of $Y$ is

g (y) = {\begin{cases} \frac{v^{r} y^{r - 1}}{Γ (r)} e^{- v y}, & 0 \leq y < \infty \\ 0 & otherwise \end{cases}

Γ (r) = \int_{0}^{\infty} y^{r - 1} e^{- y} d y

original definition:

g (y) = [\frac{1}{Γ (α) β^{α}}] y^{α - 1} e^{- y / β}, 0 < y < \infty

\begin{aligned} E [y] & = \frac{v^{r}}{Γ (r)} \int_{0}^{\infty} y \cdot y^{r - 1} e^{- v y} d y \\ = \frac{v^{r}}{Γ (r)} \int_{0}^{\infty} y^{(r + 1) - 1} e^{- v y} d y \\ = \frac{v^{r}}{Γ (r)} \frac{Γ (r + 1)}{v^{r + 1}} \int_{0}^{\infty} \frac{v^{r + 1}}{Γ (r + 1)} y^{(r + 1) - 1} e^{- v y} d y \\ = \frac{r}{v} \end{aligned}

\begin{aligned} V a r (y) & = E (y^{2}) - [E (y)]^{2} \\ E (y^{2}) & = \int_{0}^{\infty} y^{2} g (y) d y = \int_{0}^{\infty} \frac{y^{2} v^{r}}{Γ (r)} y^{r - 1} e^{- v y} d y \\ = \frac{v^{r}}{Γ (r)} \int_{0}^{\infty} y^{(r + 2) - 1} e^{- v y} d y \\ = \frac{v^{r}}{Γ (r)} \frac{Γ (r + 2)}{v^{r + 2}} \int_{0}^{\infty} \frac{v^{r + 2}}{Γ (r + 2)} y^{(r + 2) - 1} e^{- v y} d y \\ = \frac{r^{2} + r}{v^{2}} \end{aligned}

⟹ V a r (y) = \frac{r}{v^{2}}

Parameter: $μ$
Likelihood: $p (y | μ)$ is Poisson
Prior: $p (μ)$ is Gamma $(r, v)$
Posterior:

Single Observation:

\begin{aligned} p (y) & = \int_{0}^{\infty} \frac{1}{y!} \frac{v^{r}}{Γ (r)} μ^{(y + r) - 1} e^{- (v + 1) μ} d μ \\ = \frac{1}{y!} \frac{v^{r}}{Γ (r)} \cdot \frac{Γ (y + r)}{(v + 1)^{(y + r)}} \int_{0}^{\infty} \frac{(v + 1)^{(y + r)}}{Γ (y + r)} μ^{y + r - 1} e^{- (v + 1) μ} d μ \end{aligned}

\begin{aligned} (Bayes’) & p (μ | y) & = \frac{p (y | μ) p (μ)}{p (y)} \\ = \frac{p (y | μ) p (μ)}{\int_{0}^{\infty} p (y | μ) p (μ) d μ} \\ = \frac{\frac{μ^{y}}{y!} e^{- μ} \frac{v^{r}}{Γ (r)} μ^{r - 1} e^{- v μ}}{\int_{0}^{\infty} \frac{1}{y!} \frac{v^{r}}{Γ (r)} μ^{(y + r) - 1} e^{- (v + 1) μ} d μ} \\ = \frac{\frac{μ^{y}}{y!} C e^{- μ} \frac{v^{r}}{Γ (r)} μ^{r - 1} e^{- v μ}}{\frac{1}{y!} \frac{v^{r}}{Γ (r)} \cdot \frac{Γ (y + r)}{(v + 1)^{(y + r)}}} \\ = \frac{(v + 1)^{y + r}}{Γ (y + r)} μ^{(y + r) - 1} e^{- (v + 1)} μ \sim Gamma (r_{*} = y + r, v_{*} = v + 1) \end{aligned}

Multiple observations:

Likelihood: $p (y_{1}, y_{2}, \dots, y_{n} | μ)$

\frac{1}{\prod_{i = 1}^{n} y_{i}!} \cdot μ^{\sum_{i = 1}^{n} y_{i}} \cdot e^{- n μ}

Posterior:*

Updating Rules

\begin{aligned} p (μ | y_{1}, y_{2}, \dots, y_{n}) & \propto p (μ) \cdot p (y_{1}, y_{2}, \dots, y_{n} | μ) \\ (independent) & \propto \frac{v^{r}}{Γ (r)} μ^{r - 1} e^{- v μ} \cdot \prod_{i = 1}^{n} \frac{μ^{y_{i}} e^{- μ}}{y_{i!}} \\ \propto μ^{(r + \sum_{i = 1}^{n} y_{i}) - 1} e^{- (v + n) μ} \\ (updating rules) & \sim Gamma (r_{*} = r + \sum_{}^{} y_{i}, v_{*} = v + n) \end{aligned}

\begin{aligned} p (μ | {y_{i}}^{n}) & = \frac{(\frac{1}{\prod_{i = 1}^{n} y_{i}!} \cdot μ^{\sum_{i = 1}^{n} y_{i}} \cdot e^{- n μ}) \cdot \frac{v^{r}}{Γ (r)} μ^{r - 1} e^{- v μ}}{\int_{0}^{\infty} \frac{1}{\prod_{}^{} y_{i}!} \frac{v^{r}}{Γ (r)} μ^{(\sum_{}^{} y_{i}) + r - 1} e^{- (n + v) μ} d μ} \end{aligned}

\begin{aligned} \int_{0}^{\infty} \frac{1}{\prod_{}^{} y_{i}!} \frac{v^{r}}{Γ (r)} μ^{(\sum_{}^{} y_{i}) + r - 1} e^{- (n + v) μ} d μ = \frac{1}{\prod_{}^{} y_{i}!} \frac{v^{r}}{Γ (r)} \frac{Γ (\sum_{}^{} y_{i} + r)}{(n + v)^{(\sum_{}^{} y_{i} + r)}} \\ ⟹ p (μ | {y_{i}}^{n}) = \frac{(n + v)^{\sum_{}^{} y_{i} + r}}{Γ (\sum_{}^{} y_{i} + r)} μ^{(\sum_{}^{} y_{i} + r) - 1} e^{- (n + v) μ} \end{aligned}

Example 2:

Suppose we wish to estimate the number of tree seedlings in a forest. We randomly install
ten square meter plots and count the number of seedlings in each resulting in counts of
51, 47, 55, 51, 57, 55, 44, 41, 53, and 56. Assume that the number of tree seedlings per
plot follows a Poisson distribution.

a) Use a gamma prior for the Poisson parameter λ. Suppose your assessment of the
expected value for λ is 45 per plot and your assessment of the variance for λ is 9 per
plot. Find a gamma prior for λ with this mean and variance. Provide its parameters,
explicitly. Justify your answer.

\begin{aligned} λ \sim g a m m a (r = ?, v = ?) \\ want E [λ] = 45 = \frac{r}{v} \\ and v a r [λ] = 9 = \frac{r}{v^{2}} \\ ⟹ r = 45 \cdot v \\ ⟹ 9 = \frac{45}{v} ⟹ 5 = v \\ ⟹ 45 = \frac{r}{5} \\ ⟹ 225 = r \end{aligned}

b) Find the posterior distribution of λ. Provide its parameters, explicitly:

\begin{aligned} p (λ | {y_{i}}_{i = 1}^{10}) & \propto \frac{λ^{\sum y_{i}}}{\prod y_{i}} e^{- n λ} \cdot \frac{v^{r}}{Γ (r)} λ^{r - 1} e^{- v λ} \\ \propto λ^{(r + \sum_{}^{} y_{i}) - 1} e^{- (n + v) λ} \\ = λ^{(735) - 1} e^{- (15) λ} ⟹ λ \sim g a m m a (735, 15) \end{aligned}

c) Summarize the posterior distribution by its first two moments (i.e. mean and variance).
If you remember the formulas, write them and use them.

E [λ] = \frac{735}{15}, v a r (λ) = \frac{735}{15^{2}}

b) Perform a Bayesian test of the hypothesis H0 : λ ≥ 50 vs Ha : λ < 50 at the 5% level.
Please, show all your work.

\begin{array}{r} P [λ \geq 50 | d a t a] = 1 - P [λ < 50 | d a t a] \\ = 1 - P [λ \leq 50 | d a t a] \\ = 1 - \int_{0}^{50} \frac{15^{735}}{Γ (735)} λ^{735 - 1} e^{- 15 λ} d λ \end{array}

Normal approximation:

\begin{aligned} P [λ \geq 50] \approx P [\frac{λ - 49}{\sqrt{3.2667}} \geq \frac{50 - 49}{\sqrt{3.2667}}] = P [z \geq 0.5532] & = 1 - P [z < 0.5532] \\ \approx 1 - P [z \leq 0.55] = 0.2912 \end{aligned}

Effective Sample Size

Approach 1:
Using posterior mean for $μ$ in terms of sample mean and prior mean

\begin{aligned} E (μ | d a t a) & = \frac{r_{*}}{v_{*}} = \frac{(\sum_{}^{} y_{i} + r)}{n + v} = \frac{(\sum_{}^{} y_{i})}{n + v} + \frac{r}{n + v} \\ = \frac{n}{n + v} \frac{\sum_{}^{} y_{i}}{n} + \frac{v}{n + r} \frac{r}{v} \\ = \frac{n}{n + v} sample mean + \frac{v}{n + v} prior mean \end{aligned}

$⟹$ Effective sample size $= v$

Approach 2:
Likelihood: $p ({y_{i}}_{i = 1}^{n} | μ) = \frac{1}{\prod y_{i}!} μ^{\sum_{}^{} y_{i}} e^{- m μ}$
prior: $\frac{v^{r}}{Γ (r)} μ^{r - 1} e^{- v μ}$

effective sample size = $v$

Non-Informative Prior

Jeffreys' Prior

g (μ) \propto \frac{1}{\sqrt{μ}}

Gamma(r=1,v=0) Flat Uniform

Example 1
$p (μ)$ is Gamma $(r = 1, v = \frac{1}{10})$

E (μ) = 10, V a r (μ) = 100

Example 2
$p (μ)$ is Gamma $(r = 1, v = \frac{1}{100})$

E (μ) = 100, V a r (μ) = 10, 000

$⟹$ non-informative is Gamma( $r, lim_{n \to \infty} \frac{1}{n}$ ) $=$ Gamma $(r, v = 0)$

Becomes flat.

Posterior Predictive Distribution

r_{*} = r + \sum_{}^{} y_{i}, v_{*} = v + n

$y_{*}$ is the new observation
$y$ was our original data

\begin{aligned} \int_{0}^{\infty} p (y_{*} | λ) \cdot (λ | y) d λ \\ = \int_{0}^{\infty} \frac{λ^{y_{*}}}{y_{*}!} e^{- λ} \cdot (\frac{v_{*}^{r_{*}}}{Γ (r_{*})}) (λ^{r_{*} - 1} e^{- v_{*} λ}) d λ \\ = (\frac{v_{*}^{r_{*}}}{Γ (r_{*})}) \frac{1}{y^{*}!} \int_{0}^{\infty} λ^{(r * + y *) - 1} e^{- (v_{*} + 1) λ} d λ \\ = \frac{1}{y_{} *!} (\frac{v_{*}^{r_{*}}}{Γ (r_{*})}) \frac{Γ (y_{*} + r_{*})}{(v_{*} + 1)^{y_{*} + r_{*}}} \end{aligned}

Say $r_{*} = 1$ and $v_{*} = 1$

\begin{aligned} p (y_{*} | y) & = \frac{1}{y_{*}!} (\frac{1}{Γ (1)}) \frac{Γ (y_{* + 1})}{(2)^{y_{*} + 1}} \\ = {(\frac{1}{2})}^{y_{*}} (\frac{1}{2}) \end{aligned}

Geometric random variable. Number of failures until first success. $y_{*} + 1$ is when we have a success. discrete random variables

Example 3:

A geologist wishes to study the incidence of seismic movements in a given region. She then selects m independent but geologically similar observation points and counts the number of movements in a specific time interval. The observational model is Yi ∼ Pois(μ), where Yi , i = 1, 2, · · · , m, is the number of occurrences in the ith observation point and μ is the average rate of seismic movements.
a) From her previous experience, the researcher assumes that E (μ) = 2 movements per time interval an that V (μ) = 0.40 and uses these values to specify a conjugate prior. Find parameters of prior distribution and provide them explicitly.

\begin{aligned} \frac{r}{v} = 2 ⟹ r = 2 v \\ \frac{r}{v^{2}} = 0.4 ⟹ \frac{2}{v} = 0.4 ⟹ v = 5 \\ ⟹ r = 10, G a m m a (10, 5) \end{aligned}

Equivalent sample size $= 5$

b) Assuming that (2, 3, 0, 0, 1, 0, 2, 0, 3, 0, 1, 2) was observed. What is the posterior distribution? Find it and provide its parameters explicitly.

Update rules:
$G a m m a (r_{*} = 10 + 14, v_{*} = 5 + 12)$

c) She wishes to find the probability that the number of seismic
movements in an (m + 1)th site is 2 based on the observations
she had made, i.e., p(X = 2|y1, ..., ym). Find it.

\begin{aligned} = \int_{0}^{\infty} \frac{μ^{y_{n e w}}}{y_{n e w}!} e^{- μ} \cdot \frac{v_{*}^{r *}}{Γ (r_{*})} μ^{r_{*} - 1} e^{- v_{*} λ} d μ \\ = \frac{1}{y_{n e w}!} \cdot \frac{v_{*}^{r *}}{Γ (r_{*})} \int_{0}^{\infty} μ^{(r_{*} + y_{n e w}) - 1} e^{- (v_{*} + 1) μ} d μ \\ = \frac{1}{y_{n e w}!} \cdot \frac{v_{*}^{r *}}{Γ (r_{*})} \cdot \frac{Γ (r_{*} + y_{n e w})}{(v_{*} + 1)^{r_{*} + y_{n e w}}} \end{aligned}

sub in $r_{*} = 24, v_{*} = 17,$ and $y_{n e w} = 2$ , then solve.