2. MLE in Simple Regression Model

ε_{i} \sim N (0, σ^{2})

\begin{aligned} y_{i} \sim N o r m a l \\ E (Y_{i}) = E [β_{0} + β_{} 1 x_{i} + ε_{i}] = β_{0} + β_{1} x_{i} \\ V (Y_{i}) = V [β_{0} + β_{1} x_{i} + ε_{i}] = V (ε_{i}) = σ^{2} \end{aligned}

Likelihood for a single observation:

L (y_{i} | β_{0}, β_{1}, σ^{2}) = \frac{1}{\sqrt{2 π σ}} e^{(y_{i} - β_{0} - β_{1} x_{i})^{2} / 2 σ^{2}} = \frac{1}{\sqrt{2 π σ}} e^{- ε_{i}^{2} / 2 σ^{2}} = L (ε_{i})

Joint density for $ε_{1}, ε_{2}, \dots, ε_{n}$ or $y_{1}, y_{2}, \dots, y_{n}$ :

(\frac{1}{\sqrt{2 π σ}})^{n} e^{\sum_{i = 1}^{n} - ε_{i}^{2} / 2 σ^{2}}

Note: which is minimized when we minimize

\sum_{i = 1}^{n} ε_{i}^{2}

Log-Likelihood
Note: can be used for $σ^{2}, β_{1}, β_{0}$ . Take partial wrt $σ^{2}$ not $σ .$

\begin{array}{r} ℓ = \frac{n}{2} \ln [\frac{1}{2 π σ^{2}}] - \frac{1}{2 σ^{2}} \sum_{}^{} [y_{i} - β_{0} - β_{1} x_{i}]^{2} \\ \frac{\partial ℓ}{\partial β_{0}} = \frac{1}{σ^{2}} \sum_{}^{} [y_{i} - β_{0} - β_{1} x_{i}] \overset{want}{=} 0 \\ \frac{\partial ℓ}{\partial β_{1}} = \frac{1}{σ^{2}} \sum_{}^{} [y_{i} - β_{0} - β_{1} x_{i}] x_{i} \overset{want}{=} 0 \\ \frac{\partial ℓ}{\partial σ^{2}} = \frac{- n}{2 σ^{2}} + \frac{1}{2 σ^{4}} \sum_{}^{} [ε_{i}]^{2} \overset{w a n t}{=} 0 \end{array}

which gives us normal equations
So the least-squares estimator is the MLE

$N o t e :$

\begin{aligned} ℓ & = \frac{n}{2} \ln [\frac{1}{2 π σ^{2}}] - \frac{1}{2 σ^{2}} \sum_{}^{} [y_{i} - β_{0} - β_{1} x_{i}]^{2} \\ = \frac{n}{2} \ln [1] - \frac{n}{2} \ln [2 π σ^{2}] - (σ^{2})^{- 1} \frac{1}{2} \sum_{}^{} [y_{i} - β_{0} - β_{1} x_{i}]^{2} \end{aligned}

After differentiation we get:

σ^{2} = \frac{\sum_{}^{} ε^{2}}{n} = \frac{S S E}{n}

Biased estimate. Unbiased $\frac{S S E}{n - 2}$

Gauss-Markov Theorem

If the error terms in a linear regression model are uncorrelated, have equal variances, and the expectation of 0 (homoscedastic) then the ordinary least squares estimator has the lowest sampling variance within the class of linear unbiased estimators.

\begin{aligned} E [ε_{i}] = 0 \\ V a r (ε_{i}) = σ^{2} < \infty \forall i \\ C o v (ε_{i}, ε_{j}) = 0, \forall i \neq j \end{aligned}