5. SLR Confidence & Prediction Intervals

Linear Function of Parameters

Model: $y_{i} = β_{0} + β_{1} (x_{i} - \bar{x}) + ε_{i}, ε_{i} \sim N (0, σ^{2})$
Target: Estimating $E (y_{i})$ at $x_{i}$

E (y_{i}) = β_{0} + β_{1} (x_{i} - \bar{x})

Estimator: $E ({\hat{y}}_{i})) = {\hat{β}}_{0} + {\hat{β}}_{1} (x_{i} - \bar{x})$

(or Target: $θ = a_{0} β_{0} + a_{1} β_{1}$ , Estimator: $\hat{θ} = a_{0} {\hat{β}}_{0} + a_{1} {\hat{β}}_{1}$ )

We know ${\hat{β}}_{0} \sim$ Normal and ${\hat{β}}_{1} \sim$ Normal

\begin{aligned} E [{\hat{β}}_{0}] = E [\bar{y}] = \frac{1}{n} \sum_{}^{} E [y_{i}] & = \frac{1}{n} \sum_{}^{} β_{0} + β_{1} (x_{i} - \bar{x}) \\ = \frac{n β_{0}}{n} + \frac{β_{1}}{n} \sum_{}^{} {(x_{i} - \bar{x})}^{0} \\ = β_{0} \end{aligned}

\begin{array}{r} V a r [{\hat{β}}_{0}] = V a r [\frac{1}{n} \sum_{}^{} y_{i}] = \frac{1}{n^{2}} n σ^{2} = \frac{σ^{2}}{n} \end{array}

\begin{aligned} E [{\hat{β}}_{1}] & = \sum_{}^{} E [\frac{y_{i} (x_{i} - \bar{x})}{S_{X X}}] \\ = \sum_{}^{} (\frac{x_{i} - \bar{x}}{S_{X X}}) E [y_{i}] \\ = \sum_{}^{} (\frac{x_{i} - \bar{x}}{S_{X X}}) (β_{0} + β_{1} x_{i}) \\ = \frac{β_{0}}{S_{X X}} \sum_{}^{} {(x_{i} - \bar{x})}^{= 0 by (1)} + \frac{β_{1}}{S_{X X}} \sum_{}^{} (x_{i} - \bar{x}) x_{i} \\ = \frac{β_{1}}{S_{X X}} \sum_{}^{} x_{i}^{2} - n {\bar{x}}^{2} \\ (unbiased) & = β_{1} \end{aligned}

\begin{aligned} V a r [{\hat{β}}_{1}] & = σ^{2} \sum_{}^{} {[\frac{x_{i} - \bar{x}}{S_{X X}}]}^{2} \\ = \frac{σ^{2}}{S_{X X}^{2}} \sum_{}^{} (x_{i} - \bar{x})^{2} \\ = \frac{σ^{2}}{S_{X X}} \end{aligned}

Distribution of $\hat{θ}$ :

$\hat{θ}$ is a linear combination of Normals and therefore it is Normally distributed.

E [\hat{θ}] = a_{0} E [{\hat{β}}_{0}] + a_{1} E [{\hat{β}}_{1}] = a_{0} β_{0} + a_{1} β_{1} = θ

\begin{aligned} V a r [\hat{θ}] = V a r [a_{0} {\hat{β}}_{0} + a_{1} {\hat{β}}_{1}] & = C o v (a_{0} {\hat{β}}_{0} + a_{1} {\hat{β}}_{1}, a_{0} {\hat{β}}_{0} + a_{1} {\hat{β}}_{1}) \\ (1) & = V a r (a_{0} {\hat{β}}_{0}) + V a r (a_{1} {\hat{β}}_{1}) + \underset{wts=0}{\underset{―}{2 a_{0} a_{1} C o v ({\hat{β}}_{0}, {\hat{β}}_{1})}} \\ = a_{0}^{2} V a r (\hat{β_{0}}) + a_{1}^{2} V a r ({\hat{β}}_{1}) \\ = a_{0}^{2} (\frac{σ^{2}}{n}) + a_{1}^{2} (\frac{σ^{2}}{S_{x x}}) = σ^{2} (\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}) \end{aligned}

Note: $C o v (x, y) = E [(x - μ_{x}) (y - μ_{y})]$ , $C o v (x, x) = V a r (x)$

C o v ({\hat{β}}_{0}, {\hat{β}}_{1}) = C o v (\sum_{}^{} \frac{1}{n} y_{i}, \sum_{}^{} \underset{= c_{i}}{(\underset{―}{\frac{x_{i} - \bar{x}}{S_{x x}})}} y_{i})

\begin{array}{cccc} \frac{1}{n} y_{1} & \frac{1}{n} y_{2} & \dots & \frac{1}{n} y_{n} \\ c_{1} y_{1} & \frac{c_{1}}{n} c o v (y_{1}, y_{1}) = \frac{c_{1} σ^{2}}{n} & \frac{c_{1}}{n} c o v (y_{2}, y_{2}) = 0 & \dots & \frac{c_{1}}{n} c o v (y_{1}, y_{n}) = 0 \\ c_{2} y_{2} & \frac{c_{2}}{n} c o v (y_{2}, y_{1}) = 0 & \frac{c_{2}}{n} c o v (y_{2}, y_{1}) = \frac{c_{2} σ^{2}}{n} & \dots & \frac{c_{2}}{n} c o v (y_{2}, y_{n}) = 0 \\ ⋮ \\ c_{n} y_{n} & \frac{c_{n}}{n} c o v (y_{n}, y_{1}) = 0 & \frac{c_{n}}{n} c o v (y_{n}, y_{2}) = 0 & \dots & \frac{c_{n}}{n} c o v (y_{n}, y_{n}) = \frac{c_{n} σ^{2}}{n} \end{array}

⟹ C o v ({\hat{β}}_{0}, {\hat{β}}_{1}) = \frac{σ^{2}}{n} \sum_{}^{} c_{i} = \frac{σ^{2}}{S_{x x} \cdot n} {\sum_{}^{} (x_{i} - \bar{x})}^{0} = 0

⟹ \frac{\hat{θ} - θ}{σ \sqrt{\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}}} \sim N (0, 1)

Also $\frac{(n - 2) s^{2}}{σ^{2}} \sim χ_{n - 2}^{2}$

⟹ \frac{\hat{θ} - θ}{s \sqrt{\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}}} \sim t_{n - 2}

note: Centered Model
$a_{0} = 1$
$a_{1} = (x_{i} - \bar{x})$

Also note that if $c o v (x, y) = 0$ then we know that $x, y$ are independent if they are normal (not always the case for other distributions.)

Confidence Interval

\hat{θ} \pm t_{\frac{α}{2}, n - 2} \cdot (s \sqrt{\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}})

P [\hat{θ} - t_{\frac{α}{2}, n - 2} \cdot (s \sqrt{\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}}) \leq θ \leq \hat{θ} + t_{\frac{α}{2}, n - 2} \cdot (s \sqrt{\frac{a_{0}^{2}}{n} + \frac{a_{1}^{2}}{S_{x x}}})] = 1 - α

for $β_{1}, a_{0} = 0, a_{1} = 1$
for $β_{0}, a_{0} = 1, a_{1} = 0$

Prediction Interval

Model:

y_{i} = β_{0} + β_{1} (x_{i} - \bar{x}) + ε_{i}

Initial Dataset:

(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{n}, y_{n})

Estimators: ${\hat{β}}_{0} = \bar{y}$ ${\hat{β}}_{1} = \frac{S_{x y}}{S_{x x}}$

Target: $y_{*} = β_{0} + β_{1} (x_{*} - \bar{x}) + ε_{*}$ , $ε_{*} \sim N (0, 1)$

Prediction: ${\hat{y}}_{*} = {\hat{β}}_{0} + {\hat{β}}_{1} (x_{*} - \bar{x})$

$y_{*}$ is random and independent of our original dataset. $y_{*} \sim$ Normal
$E (y_{*}) = β_{0} + β_{1} (x_{*} - \bar{x})$
$V a r (y_{*}) = σ^{2}$

${\hat{y}}_{*}$ is a linear combination of Normals
$E [{\hat{y}}_{*}] = β_{0} + β_{1} (x_{*} - \bar{x})$
$V a r [{\hat{y}}_{*}] = \frac{σ^{2}}{n} + \frac{(x_{*} - \bar{x}) σ^{2}}{S_{x x}}$

$(y_{*} - {\hat{y}}_{*}) \sim$ Normal
$E [y_{*} - {\hat{y}}_{*}] = β_{0} + β_{1} (x_{*} - \bar{x}) - β_{0} - β_{1} (x_{*} - \bar{x}) = 0$
$V a r [y_{*} - {\hat{y}}_{*}] = σ^{2} [1 + \frac{1}{n} + \frac{(x_{*} - \bar{x})^{2}}{S_{x x}}]$

$y_{*} = {\hat{y}}_{*} + \underset{prediction error}{(y_{*} - \hat{y_{*}})}$

⟹ \frac{y_{*} - {\hat{y}}_{*}}{σ \sqrt{1 + \frac{1}{n} + \frac{(x_{*} - \bar{x})^{2}}{S_{x x}}}} \sim N (0, 1)

⟹ \frac{y_{*} - {\hat{y}}_{*}}{s \sqrt{[1 + \frac{1}{n} + \frac{(x_{*} - \bar{x})^{2}}{S_{x x}}]}} \sim χ_{n - 2}^{2}

{\hat{y}}_{*} \pm t_{\frac{α}{2}, n - 2} \cdot (s \sqrt{1 + \frac{1}{n} + \frac{(x_{*} - \bar{x})^{2}}{S_{x x}}})