2. Bayes Inference for Discrete Random Variables

Bayes' Universe and Table

Lets say we would like to guess # of red balls := N given the amount observed in a sample := y

Bayesian Universe: 2-dimensional (N,y)

Vertical dimension: possible values for parameter (unobservable)
Horizontal dimension: sample values (observable)

\begin{array}{ccccccc} prior & y_{1} & \dots & y_{j} & \dots & y_{J} \\ x_{1} & g (x_{1}) & f (x_{1}, y_{1}) & \dots & f (x_{1}, y_{j}) & \dots & f (x_{1}, y_{J}) \\ ⋮ & ⋮ & ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{i} & g (x_{i}) & f (x_{i}, y_{1}) & \dots & f (x_{i}, y_{j}) & \dots & f (x_{i}, y_{J}) \\ ⋮ & ⋮ & ⋮ & ⋱ & ⋮ & ⋱ & ⋮ \\ x_{I} & g (x_{I}) & f (x_{I}, y_{1}) & \dots & f (x_{I}, y_{j}) & \dots & f (x_{I}, y_{J}) \\ f (y_{1}) & \dots & f (y_{j}) & \dots & f (y_{J}) \end{array}

Reduced after seeing $Y = y_{j}$

\begin{array}{ccccccccc} \cdot & \cdot & \cdot & \cdot & (x_{1}, y_{j}) & \cdot & \cdot & \cdot & \cdot \\ \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot \\ \cdot & \cdot & \cdot & \cdot & (x_{i}, y_{j}) & \cdot & \cdot & \cdot & \cdot \\ \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot & \cdot \\ \cdot & \cdot & \cdot & \cdot & (x_{I}, y_{j}) & \cdot & \cdot & \cdot & \cdot \end{array}

$D$ is observed data
$A_{i}$ are partitions

\begin{array}{lcccl} Scenario & Prior P (A_{i}) & Likelihood P (D | A_{i}) & Joint P (D \cap A_{i}) & Posterior P (A_{i} | D) \\ A_{1} & P (A_{1}) & P (D | A_{1}) & P (D \cap A_{1}) = P (D | A_{1}) P (A_{1}) & \frac{P (D | A_{1}) P (A_{1})}{P (D)} \\ A_{2} & P (A_{2}) & P (D | A_{2}) & P (D \cap A_{2}) = P (D | A_{2}) P (A_{2}) & \frac{P (D | A_{2}) P (A_{2})}{P (D)} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ A_{n} & P (A_{n}) & P (D | A_{n}) & P (D \cap A_{n}) = P (D | A_{n}) P (A_{n}) & \frac{P (D | A_{n}) P (A_{n})}{P (D)} \\ 1 & \sum_{i = 1}^{n} P (D \cap A_{i}) = P (D) & 1 \end{array}

Before new observations our Posteriors become our Priors

For Multiple Trials Which We Can Classify as Success or Failure:
We use the binomial distribution

f (x) = (\binom{n}{x}) p^{x} (1 - p)^{n - x}

(\binom{n}{x}) = \frac{n!}{x! \times (n - x)!}

to calculate our likelihood.

Parameter of interest: Proportion ( $π$ )
Prior Distribution: Discrete probability distributions
Likelihood: Binomial
Posterior Distribution: Discrete probability distribution using Bayes' theorem

Expectation

$A_{i}$ are the outcomes that we are modelling

\begin{array}{r} \sum_{i = 1}^{n} "Model" \cdot Posterior = \sum_{i = 1}^{n} A_{i} \cdot P (A_{i} | D) \end{array}