1. Simple Linear Regression

3 Dataset: $(x_{1}, y_{1}), \dots (x_{n}, y_{n})$
$x =$ predictor variable (assumed to be constant)
(only one if simple linear regression otherwise a vector for multiple linear regression)

$y =$ response variable (assumed to be random/stochastic)

Assumption: response variables are normally distributed (conditioned on the predictor), variance is equal across all predictors.

The data values Y come from different conditional distributions for each $x \in X$ :

Y | X = x \sim p (y | x)

Goal: Find the line that best fits our data

{\hat{y}}_{i} = {\hat{β}}_{0} + {\hat{β}}_{1} x_{i}

Y_{i} | X = y_{i} \sim N (β_{0} + β_{1} x_{i}, σ^{2})

Carry out a hypothesis test:

\begin{array}{r} H_{0} : β_{1} = 0 \\ H_{a} : β_{1} \neq 0 \end{array}

If reject $H_{0} ⟹$ construct C.I for $E (y)$ at $x = x^{*}$ , also want to predict future observation ( $y^{*}$ ) when $x = x^{*}$

\begin{array}{ll} When p (y ∣ x) is... & Then you have... \\ Normal & Classical regression \\ Poisson & Poisson regression \\ Negative binomial & Negative binomial regression \\ Bernoulli & Logistic regression \\ Beta & Beta regression \\ Lognormal & Lognormal regression \\ ⋮ & ⋮ \end{array}

Main Assumptions

Randomness (wrt Y)
Correct Functional Specification (linear, polynomial, etc.)
L: Expected value is given by a linear function
Uncorrelated Errors
I: Independent
N: Normally Distributed
E: Equal variance
Constant variance (Y|X)
Normality

y_{i} = β_{0} + β_{1} x_{i} + ε_{i}

ε_{i} \overset{i i d}{\sim} N (0, σ^{2})

(homoscedasticity)

Pasted image 20250505132418.webp|433

Pasted image 20250505132454.webp|434

Least Squares Point Estimates

$E (Y) = β_{0} + β_{1} x$

Wants to fit a line to $(x_{1}, y_{1}), \dots, (x_{n}, y_{n})$

min_{β_{0}, β_{1}} Q = \sum_{i = 1}^{n} [y_{i} - (β_{0} + β_{1} x_{i})]^{2}

Where $Q$ is the sum of squares for error, SSE

\begin{array}{r} \frac{\partial Q}{\partial β_{0}} = - \sum_{i = 1}^{n} 2 [y_{i} - (β_{0} + β_{1} x_{i})] \\ \frac{\partial Q}{\partial β} = - \sum_{i = 1}^{n} 2 x_{i} [y_{i} - (β_{0} + β_{1} x_{i}] \end{array}

For ${\hat{β}}_{0}$

\begin{aligned} 0 & = \sum_{}^{} (y_{i} - β_{0} - β_{1} x_{i}) \\ (1) & = \sum_{}^{} y_{i} - n β_{0} - n β_{1} \bar{x} \\ = n \bar{y} - n β_{0} - n β_{1} \bar{x} \\ * & β_{0} & = \bar{y} - β_{1} \bar{x} \end{aligned}

For ${\hat{β}}_{1}$
Note: Identity $\sum_{}^{} (x_{i} - \bar{x})^{2} = \sum_{}^{} x_{i}^{2} - n {\bar{x}}^{2}$

\begin{array}{r} S_{X Y} = \sum_{}^{} (x_{i} - \bar{x}) (y_{i} - \bar{y}) = \sum_{}^{} x_{i} y_{i} - n \bar{x} \bar{y} \\ S_{X X} = \sum_{}^{} (x_{i} - \bar{x})^{2} = (n - 1) \cdot s_{x}^{2} \end{array}

\begin{aligned} 0 & = \sum_{}^{} x_{i} (y_{i} - β_{0} - β_{1} x_{i}) \\ (2) & = \sum_{}^{} x_{i} y_{i} - β_{0} \sum_{}^{} x_{i} - β_{1} \sum_{}^{} x_{i}^{2} \\ = \sum_{}^{} x_{i} y_{i} - (\bar{y} - β_{1} \bar{x}) n \bar{x} - β_{1} \sum_{}^{} x_{i}^{2} \\ = \sum_{}^{} x_{i} y_{i} - \bar{y} n \bar{x} + β_{1} n {\bar{x}}^{2} - β_{1} \sum_{}^{} x_{i}^{2} \\ = \sum_{}^{} x_{i} y_{i} - \bar{y} n \bar{x} + β_{1} (n {\bar{x}}^{2} - \sum_{}^{} x_{i}^{2}) \\ β_{1} (\sum_{}^{} x_{i}^{2} - n {\bar{x}}^{2}) = \sum_{}^{} x_{i} y_{i} - n \bar{y} \bar{x} \\ β_{1} (\sum_{}^{} (x_{i} - \bar{x})^{2}) = \sum_{}^{} (x_{i} - \bar{x}) (y_{i} - \bar{y}) \\ * & β_{1} = \frac{S_{X Y}}{S_{X X}} \end{aligned}

Normal Equations:

\begin{array}{r} (1) & n β_{0} + β_{1} \sum_{}^{} x_{i} = \sum_{}^{} y_{i} \\ (2) & β_{0} \sum_{}^{} x_{i} + β_{1} \sum_{}^{} x_{i}^{2} = \sum_{}^{} x_{i} y_{i} \end{array}

Using Cramer's Rule:

\begin{array}{r} β_{0} = \frac{Δ β_{0}}{Δ} = \frac{det [\begin{array}{c} \sum_{}^{} y_{i} & \sum_{}^{} x_{i} \\ \sum_{}^{} x_{i} y_{i} & \sum_{}^{} x_{i^{2}} \end{array}]}{d e t [\begin{array}{c} n & \sum_{}^{} x_{i} \\ \sum_{}^{} x_{i} & \sum_{}^{} x_{i^{2}} \end{array}]} = \frac{\sum_{}^{} y i \sum_{}^{} x_{i}^{2} - (\sum_{}^{} x_{i} \sum_{}^{} x_{i} y_{i})}{n \sum_{}^{} x_{i}^{2} - {[\sum_{}^{} x_{i}]}^{2}} \end{array}

\begin{array}{r} β_{1} = \frac{Δ β_{1}}{Δ} = \frac{det [\begin{array}{c} n & \sum_{}^{} y_{i} \\ \sum_{}^{} x_{i} & \sum_{}^{} x_{i} y_{i} \end{array}]}{n \sum_{}^{} x_{i}^{2} - {[\sum_{}^{} x_{i}]}^{2}} \end{array}

These are also the MLE

Correlation Coefficient

r = \frac{s_{x y}}{s_{x} s_{y}} = \frac{\frac{1}{n - 1} \sum_{i = 1}^{n} (x_{i} - \bar{x}) (y_{i} - \bar{y})}{\sqrt{\frac{1}{n - 1} \sum_{i = 1}^{n} (x_{i} - \bar{x})^{2}} \sqrt{\frac{1}{n - 1} \sum_{i = 1}^{n} (y_{i} - \bar{y})^{2}}} = \frac{\sum_{i = 1}^{n} x_{i} y_{i} - n \bar{x} \bar{y}}{\sqrt{(\sum_{i = 1}^{n} x_{i}^{2} - n {\bar{x}}^{2}) (\sum_{i = 1}^{n} y_{i}^{2} - n {\bar{y}}^{2})}}

$= \frac{C o v (x, y)}{s_{x} s_{y}}$

= {\hat{β}}_{1} \cdot \frac{\sqrt{S_{x x}}}{\sqrt{S_{y y}}}