5. Bayesian Inference on Normal Distribution

Normal Likelihood

Likelihood:

\begin{aligned} f (y | μ) & = \frac{1}{\sqrt{2 π σ^{2}}} e^{\frac{- 1}{2} {(\frac{y - μ}{σ})}^{2}} \\ f ({y_{i}}^{n} | μ, σ^{2}) & = {(\frac{1}{\sqrt{2 π σ^{2}}})}^{n} \cdot e^{\frac{- 1}{2 σ^{2}} \sum_{}^{} (y_{i} - μ)^{2}} \end{aligned}

$\bar{y} \sim N (μ, \frac{σ^{2}}{n})$

\begin{aligned} \sum_{i = 1}^{n} (y_{i} - μ)^{2} = \sum_{}^{} (y_{i} - \bar{y} + \bar{y} - μ)^{2} & = \sum_{}^{} ((y_{i} - \bar{y})^{2} + (\bar{y} - μ)^{2} + 2 (y_{i} - \bar{y}) (\bar{y} - μ)) \\ = \sum_{}^{} (y_{i} - \bar{y})^{2} + n (\bar{y} - μ)^{2} + 2 (\bar{y} - μ) {\sum_{}^{} (y_{i} - \bar{y})}^{0} \\ = \sum_{}^{} (y_{i} - \bar{y})^{2} + n (\bar{y} - μ)^{2} \end{aligned}

\begin{aligned} ⟹ L & = {[\frac{1}{2 π σ^{2}}]}^{n / 2} e^{\frac{- 1}{2 σ^{2}} [\sum_{}^{} (y_{i} - \bar{y})^{2} + n (\bar{y} - μ)^{2}]} \\ = {[\frac{1}{2 π σ^{2}}]}^{n / 2} \underset{constant wrt μ}{\underset{⏟}{e^{- 1 / (2 σ^{2}) \sum_{}^{} (y_{i} - \bar{y})^{2}}}} \cdot e^{[- 1 / (2 σ^{2})] n (\bar{y} - μ)^{2}} \end{aligned}

Assuming that $σ^{2}$ is known

f ({y_{i}} | μ) \propto e^{- n / (2 σ^{2}) (\bar{y} - μ)^{2}} \propto e^{- \frac{1}{2 (\frac{σ^{2}}{n})} (\bar{y} - μ)^{2}}

proportional to a Normal with mean $μ$ and variance $\frac{σ^{2}}{n}$ .

Thus,

p (y_{1}, y_{2}, \dots, y_{n}) \propto p (\bar{y} | μ) is Normal

Example

Suppose we take a random sample of four observations from a Normal distribution having mean μ and known variance σ2 = 1.
The observations are 3.2, 2.2, 3.6, and 4.1. The possible value of μ are 2.0, 2.5, 3.0, 3.5, and 4.0. We will use a prior that gives them all equal weight. We want to use Bayes’ Theorem to find our posterior belief about μ given the whole random sample.

x=c(3.2, 2.2, 3.6, 4.1);  
mu = c(2, 2.5, 3, 3.5, 4);  
mu.prior = rep(1/5, 5);  ## [1/5,1/5,1/5,1/5,1/5]
likelihood = dnorm(mean(x),mean=mu, sd = 1/sqrt(4));  
posterior = mu.prior*likelihood/sum(mu.prior*likelihood);  
posterior;  
## [1] 0.01579107 0.12266347 0.35052941 0.36850143  
## [5] 0.14251462

Flat Prior

Take $lim_{σ^{2} \to \infty}$

g (μ) = 1

Which is not a proper distribution so it is improper, however the posterior will integrate to 1 thus it will be proper.

Posterior

Parameter: $μ$
Likelihood: $P (y_{1}, y_{2}, \dots, y_{n} | μ, σ^{2})$
Assumption: $σ^{2}$ is known

p ({y_{i}} | μ) \propto p (\bar{y} | μ) \sim N (μ, \frac{σ^{2}}{n})

\begin{aligned} p (μ | \bar{y}) & \propto p (\bar{y} | μ) p (μ) \\ \propto e^{\frac{- 1}{2 (σ^{2} / n)} [\bar{y} - μ]^{2}} \\ \propto e^{\frac{- 1}{2 (σ^{2} / n)} [μ - \bar{y}]^{2}} \end{aligned}

⟹ p (μ | {y_{i}}) \sim N (\bar{y}, \frac{σ^{2}}{n})

Normal Prior Single Observation

Parameter: $μ$
Likelihood: $P (y_{1}, y_{2}, \dots, y_{n} | μ, σ^{2}) \propto p (\bar{y}, μ) \sim N (μ, {\frac{σ}{n}}^{2})$
Assumption: $σ^{2}$ is known
Prior: $p (μ) \sim N (μ = m, σ^{2} = s^{2})$

f (μ) = \frac{1}{\sqrt{2 π s^{2}}} e^{\frac{- 1}{2} {(\frac{μ - m}{s})}^{2}}

Posterior

\begin{aligned} p (μ | \bar{y}) & \propto p (\bar{y} | μ) p (μ) \\ \propto e^{\frac{- 1}{2 (σ^{2} / n)} [\bar{y} - μ]^{2}} \cdot e^{\frac{- 1}{2 \cdot s^{2}} {(μ - m)}^{2}} \end{aligned}

One Observation:
Parameter: $μ$
Likelihood: $P (y | μ) \sim N (μ, {\frac{σ}{n}}^{2})$
Assumption: $σ^{2}$ is known
Prior: $p (μ) \sim N (μ = m, σ^{2} = s^{2})$ $⋆$

\begin{aligned} p (μ | \bar{y}) & \propto p (y | μ) p (μ) \\ \propto e^{\frac{- 1}{2 (σ^{2})} [y - μ]^{2}} \cdot e^{\frac{- 1}{2 \cdot s^{2}} {(μ - m)}^{2}} \\ \propto e^{\frac{- 1}{2} {[\frac{(y - μ)^{2}}{σ^{2}} + \frac{(μ - m)^{2}}{s^{2}}]}^{2}} \\ \propto \exp (\frac{- 1}{2} \frac{μ^{2} (s^{2} + σ^{2}) - 2 μ (s^{2} y + σ^{2} m) + (s^{2} y^{2} + σ^{2} m)}{σ^{2} s^{2}}) \\ \propto \exp (- \frac{1}{2 σ^{2} s^{2}} [μ^{2} (s^{2} + σ^{2}) - 2 μ (s^{2} y + σ^{2} m)]) \\ \cdot \exp (\frac{1}{2 σ^{2} s^{2}} (s^{2} y^{2} + σ^{2} m^{2})) \\ \propto \exp (- \frac{s^{2} + σ^{2}}{2 σ^{2} s^{2}}) (μ^{2} - \frac{2 μ (s^{2} y + σ^{2} m)}{s^{2} + σ^{2}}) \end{aligned}

[μ - \frac{s^{2} y^{2} + σ^{2 m}}{s^{2} + σ^{2}}] = μ^{2} - \frac{2 μ (s^{2} y + σ^{2} m)}{s^{2} + σ^{2}} + [\frac{s^{2} y^{2} + σ^{2 m}}{s^{2} + σ^{2}}]^{2}

\begin{aligned} p (μ | y) & \propto \exp (- \frac{s^{2} + σ^{2}}{2 σ^{2} s^{2}}) {[μ - \frac{s^{2} y + σ^{2} m}{s^{2} + σ^{2}}]}^{2} \\ \propto \exp (- \frac{1}{2 \frac{σ^{2} s^{2}}{σ^{2} + s^{2}}} {(μ - \frac{σ^{2} m + s^{2} y}{σ^{2} + s^{2}})}^{2}) \end{aligned}

mean $\frac{s^{2} y + σ^{2} m}{s^{2} + σ^{2}} = m_{*}$ variance $\frac{σ^{2} s^{2}}{s^{2} + σ^{2}} = s_{*}^{2}$

Updating Rules

Precision: 1/Variance $⟹$ posterior precision $\frac{s^{2} + σ^{2}}{σ^{2} s^{2}}$

\frac{1}{s_{*}^{2}} = \frac{s^{2}}{σ^{2} s^{2}} + \frac{σ^{2}}{σ^{2} s^{2}} = \frac{1}{σ^{2}} + \frac{1}{s^{2}}

Posterior Precision = Sample Precision + Prior Precision

m_{*} = \frac{s^{2} y}{s^{2} + σ^{2}} + \frac{σ^{2} m}{s^{2} + σ^{2}} = \frac{\frac{\frac{s^{2} y}{σ^{2} s^{2}}}{s^{2} + σ^{2}}}{σ^{2} s^{2}} + \frac{\frac{\frac{σ^{2} m}{σ^{2} s^{2}}}{s^{2} + σ^{2}}}{σ^{2} s^{2}} = [\frac{\frac{1}{σ^{2}}}{\frac{1}{s_{*}^{2}}}] \bar{y} + [\frac{\frac{1}{s^{2}}}{\frac{1}{s_{*}^{2}}}] m

Posterior Mean = \frac{Sample Precision}{Posterior Precision} (Sample Mean) + \frac{Prior Precision}{Posterior Precision} (Prior Mean)

Normal Prior Multiple Observations

Parameter: $μ$
Likelihood: $p ({y_{i}} | μ)$ is $N (μ, σ^{2})$
Prior: $p (μ) \sim N (m, s^{2})$
Posterior: $p (μ | {y_{i}})$

p ({y_{i}} | μ) \propto p (\bar{y} | μ) \sim N (μ, \frac{σ^{2}}{n})

⟹ \frac{1}{s_{*}^{2}} = \frac{n}{σ^{2}} + \frac{1}{s^{2}}

⟹ m_{*} = [\frac{\frac{n}{σ^{2}}}{\frac{1}{s_{*}^{2}}}] \bar{y} + [\frac{\frac{1}{s^{2}}}{\frac{1}{s_{*}^{2}}}] m

Equivalent Sample Size

Prior Variance = s^{2} = \frac{σ^{2}}{n_{equiv}}

\begin{array}{r} m_{*} = [\frac{n (\frac{1}{σ^{2}})}{\frac{1}{s_{*}^{2}}}] \bar{y} + [\frac{\frac{σ^{2}}{s^{2}} (\frac{1}{σ^{2}})}{\frac{1}{s_{*}^{2}}}] m \end{array}

Example

Arnie and Barb are going to estimate the mean length of one-year-old rainbow trout in a stream. Previous studies in other streams have shown the length of yearling rainbow trout to be Normally distributed with known standard deviation of 2 cm. Arnie decides his prior mean is 30 cm. He decides that he doesn’t believe it is possible for a yearling rainbow to be less than 18 cm or greater than 42 cm. Thus his prior standard deviation is 4 cm. Thus he will use a Normal(30, 4) prior. Barb doesn’t know anything about trout, so she decides to use the “flat” prior.

They take a random sample of 12 yearling trout from the stream and find the sample mean ̄y = 32 cm. Arnie and Barb find their posterior distributions using the simple updating rules for the Normal conjugate family.

$σ^{2} = 4$ by the empirical rule
$m = 35$
$n = 12$
$\bar{y} = 32$

\frac{1}{s_{*}^{2}} = \frac{12}{4} + \frac{1}{4^{2}} ⟹ s_{*}^{2} = \frac{49}{16}

m_{*} = \frac{\frac{49}{16}}{\frac{49}{16}} \cdot 32 + \frac{\frac{1}{16}}{\frac{49}{16}} \cdot 30 = \underset{convex combination}{\underset{⏟}{[[32 ∥ \frac{48}{49}]] + [[30 ∥ \frac{1}{49}]]}}

Example

The standard process for making a polymer has mean yield 35%. A
chemical engineer has developed a modified process. She runs the
process on 10 batches and measures the yield (in percent) for each
batch. They are:
38.7 40.4 37.2 36.6 35.9
34.7 37.6 35.1 37.5 35.6
Assume that yield is Normal(μ, σ2) where the standard deviation
σ = 3 is known.

\frac{1}{s_{*}^{2}} = \frac{1009}{900}

m_{*} = \frac{\frac{\frac{100}{900}}{1009}}{900} (36.93) + \frac{\frac{9}{900}}{\frac{1009}{900}} 30

Example

Of those women who are diagnosed to have early-stage breast cancer, one-third eventually die of the disease. Suppose a community public health department instituted a screening program to provide for the early detection of breast cancer and to increase the survival rate π of those diagnosed to have the disease. A random sample of 27 women was selected from among those who were periodically screened by the program and who were diagnosed to have the disease. Let y represent the number of those in the sample who survive the disease.
Answer each of the following questions. You have to show all your work to get full credit.

a) If you wish to detect whether the community screening program has been effective, state the null hypothesis that should be tested.

b) State the alternative hypothesis.

$H_{0} : π \leq 2 / 3, H_{a} : π > 2 / 3$

c) If 20 women in the sample of 27 survive the disease, find the posterior distribution of π. Use a Beta(2, 1) prior for π. Provide parameters of posterior distribution, explicitly.

p (π | Data) \propto (π^{20} (1 - π^{7})) \cdot (π^{2 - 1} (π - 1)^{1 - 1})

⟹ π | Data \sim B e t a (22, 8)

d) Using parts a), b) and c), can you conclude that the community screening program was
effective? Test at the 5% level of significance in a Bayesian manner. Show all your work
and explain the practical conclusions from your test.

Want $P [π \leq 2 / 3 | Data]$ using normal approximation we have $P [z < \frac{(2 / 3 - \frac{22}{30})}{\sqrt{\frac{22 \cdot 8}{(30)^{2} 31}}}] = 0.2005 > α$ , Failed to reject $H_{0}$