Model

MLE Criterion

by derivative theorems

Key Properties for Hat Matrix

can now apply chi theorems
$(I-H)$ is also symmetric and idempotent

Expected Value of a Random Vector

Covariance Matrix of a Random Vector

Let $Y_1,Y_2$ be two random variables and ${\mu }_1=\mathbb{E}[Y_1],{\mu }_2=\mathbb{E}[Y_2],{\sigma }_1^2=V(Y_1),{\sigma }_2^2=V(Y_2),{\sigma }_{12}^2=cov(Y_1,Y_2)=\mathbb{E}[(Y_1-{\mu }_1)(Y_2-{\mu }_2)]$

Useful Properties

$A=$ matrix of constants, $c=$ vector of constants, $y=$ random vector

1. $\mathbb{E}[Ay+c]=A\mathbb{E}[y]+c$, can show easily that expectation is a Linear Transformations and this property follows.
2. $cov[Ay+c]=Acov(y)A^T$
   $=\mathbb{E}[(Ay+c-\mathbb{E}(Ay+c))(Ay+c-\mathbb{E}(Ay+c){)}^T]=\mathbb{E}[(Ay+c-(A\mathbb{E}(y)+c))(Ay+c-(A\mathbb{E}(y)+c){)}^T]$
   $=\mathbb{E}[(Ay-A\mathbb{E}(y))(Ay-A\mathbb{E}y{)}^T]=\mathbb{E}[A(y-\mathbb{E}(y))(y-\mathbb{E}(y){)}^T{A}^T]=Acov(y)A^T$

So $$cov(Y)=cov(\varepsilon+X\beta)=cov[\varepsilon]=\begin{pmatrix}
\sigma^{2} & 0 & \dots & 0 \
0 & \sigma^{2}&\dots & 0 \
\vdots & \vdots & \ddots& \vdots \
0 & 0 & \dots & \sigma^{2}
\end{pmatrix}$$
Let $W$ be a random matrix
3) $\mathbb{E}(tr(W))=tr\mathbb{E}(W)$

Model Moments

Parameters

Error

Can do $\mathbb{E}[Y-HY]=\mathbb{E}[(I-H)Y]$

$Cov(\hat{\epsilon })=(I-H)Cov(Y)(1-H{)}^T=(I-H){\sigma }^2(I-H)=(1-H){\sigma }^2$ since symmetric and idempotent

Say $B=A\mathbb{E}[\epsilon {\epsilon }^T]$, then:
$B_{ii}=\sum _j{A}_{ij}\mathbb{E}[{\epsilon }_j{\epsilon }_i]=\sum _j{A}_{ij}\mathbb{E}[{\epsilon }_i{\epsilon }_j]$

So:
$\mathrm{tr}(A\mathbb{E}[\epsilon {\epsilon }^T])=\sum _i{B}_{ii}=\sum _i\sum _j{A}_{ij}\mathbb{E}[{\epsilon }_i{\epsilon }_j]=\sum _{i,j}{A}_{ij}\mathbb{E}[{\epsilon }_i{\epsilon }_j]$

Which is exactly the same as:
$\mathbb{E}[{\epsilon }^{T}A\epsilon ]=\sum _{i,j}{A}_{ij}\mathbb{E}[{\epsilon }_i{\epsilon }_j]$

Confidence Interval

For $\mathbb{E}(y_0)$ at $x_0$
Model: $\hat{y}=X_{n\times (p+1)}^T{\hat{\beta }}_{(p+1)\times n}+{\epsilon }_n$

It can be shown that $\frac{[n-(p+1)]s^2}{{\sigma }^2}\sim {\chi }^2(df=n-(p+1))$ using theorem 16.4

Confidence Interval

Prediction Interval

Initial dataset $(Y_1,Y_2,\dots ,Y_n)$
Target $Y_{\ast }=$ new observation
$Y_{\ast }={\beta }_0+{\beta }_1{x}_{\ast 1}+\cdots +{\beta }_p{x}_{\ast p}+{\epsilon }_{\ast }=X_{\ast }^T\beta +{\epsilon }_{\ast }$
Prediction ${\hat{Y}}_{\ast }=X_{\ast }^T\beta$

$Y_{\ast }={\hat{Y}}_{\ast }+\underset{\text{prediction error}}{\underbrace{(Y_{\ast }-{\hat{Y}}_{\ast })}}$

$\mathbb{E}(Y_{\ast })=X_{\ast }^T\beta$, $cov(Y_{\ast })={\sigma }^2$ since there is only one error
$\mathbb{E}({\hat{Y}}_{\ast })=X_{\ast }^T\mathbb{E}(\hat{\beta })=X_{\ast }^T\beta$
$cov({\hat{Y}}_{\ast })=X_{\ast }^{T}cov(\hat{\beta })(X_{\ast }^T{)}^T=X_{\ast }^T{\sigma }^2(X^{T}X{)}^{-1}{X}_{\ast }={\sigma }^2\underset{1\times 1}{\underbrace{X_{\ast }^T(X^{T}X{)}^{-1}{X}_{\ast }}}=var({\hat{Y}}_{\ast })$
$\mathbb{E}[Y_{\ast }-{\hat{Y}}_{\ast }]=X_{\ast }^T\beta -X_{\ast }^T\beta =0$
$var[Y_{\ast }-{\hat{Y}}_{\ast }]\stackrel{\text{ind}}{=}{\sigma }^2+{\sigma }^2{X}_{\ast }^T(X^{T}X{)}^{-1}{X}_{\ast }$