Two Parameter Model

Derivative Theorems

Let $u=a^T$ and $x=x^{T}a$ where $a^T=(a_1,a_2,\dots ,a_p)$ is a vector of constants

If A is symmetric and $u=x^{T}Ax$ where is is all constant

Least Squares Estimate

Properties:

Residuals and Hat Matrix
$\hat{Y}=X\hat{\beta }=\underset{\text{Hat Matrix:H}}{\underbrace{X(X^{T}X{)}^{-1}{X}^T}}Y$

$\hat{\epsilon }=Y-\hat{Y}=Y-HY=(I-H)Y$

$X^T\hat{e}=X^T(I-H)Y=X^{T}Y-X^{T}HY=X^{T}Y-X^{T}Y=0⟹$ orthogonal

Residual Sum of Squares and Hat Matrix
Let $X$ have full rank $r+1\le n$. The least squares estimate of $\beta$ in the Classical Linear Regression Model is given by:

Let $\hat{y}=X\hat{\beta }=Hy$ denote the fitted values of $y$, where

is called the "hat" matrix. Then the residuals

satisfy $X^T\hat{e}=0$ and ${\hat{y}}^T\hat{e}=0$. Also, the

residual sum of squares is:

Hat Matrix

$\hat{Y}=X\hat{\beta }=\underset{\text{Hat Matrix:H}}{\underbrace{X(X^{T}X{)}^{-1}{X}^T}}Y$

Elements:

when $i=j$ then we are calculating how much weight the observed value has in its own fitted value.

$H_{22}$ Second row, second column

For SLR

For large sample sizes (>30) values 2 times the average leverage value should be considered large. For small sample sizes values 3 times the average leverage value should be considered large.

Testing Residuals

var(${\hat{\epsilon }}_i$)=${\sigma }^2(1-H_{ii})$

approximately $t_{n-(p+1)}$ where p is the number of parameters (+ slope)

where MSE is calculated without observation $i$
exactly $t_{n-(p+1)}$