9. MLR Hypothesis Testing

SLR Review

Recall our earlier F-tests for anova T tests with respect to correlation and our parameters
Where $F_{*} = \frac{M S R}{M S E} = (n - 2) \frac{r^{2}}{1 - r^{2}}$

MLR

ε_{n \times 1} \sim N_{n} (0_{n \times 1}, σ^{2} I_{n \times n})

Overall Test

y_{i} = X_{i}^{T} β + ε_{i} = β_{0} + x_{i, 1} β_{1} + x_{i, 2} β_{2} + \dots + x_{i, p} β_{p}

$H_{0} : β_{1} = β_{2} = \dots = β_{p} = 0$ , $H_{a} :$ at least one $β_{i} \neq 0$
$S S T = S S R + S S E$

R^{2} \equiv \frac{S S R}{S S T}

\begin{aligned} F_{*} & = \frac{M S R}{M S E} = \frac{\frac{S S R}{p}}{\frac{S S E}{n - (p + 1)}} = \frac{n - (p + 1)}{p} \frac{S S R}{S S E} \\ or F_{*} & = \frac{n - (p + 1)}{p} \frac{\frac{S S R}{S S T}}{\frac{S S E}{S S T}} = \frac{n - (p + 1)}{p} \frac{R^{2}}{1 - R^{2}} \end{aligned}

where df numerator = $p$ , df denominator = $n - (p + 1)$

(remember $S S R = S S Y - S S E$ )

RR = ${F_{*} such that F_{*} > F_{α, p, n - (p + 1)}}$

Addition of a Group of Variables

F_{*} = \frac{\frac{(S S E (R) - S S E (C)}{(complete parameters - reduced parameters))}}{\frac{S S E (C)}{n - complete parameters}}

R = reduced , C = complete
$H_{a} :$ At least one new parameter contributes information
RR = ${F_{*} > F_{α, C - R, n - C}}$

Example Comparing Means

Bilirubin is formed in the liver, where haemoglobin and other haemoproteines are decomposed into bile pigments. Bilirubin is partly reabsorbed by the intestine, and returns to the liver. If the liver has suffered degeneration, if the decomposition of haemoglobin is elevated, or if the gall bladder has been destroyed, bilirubin can accumulate to high levels in the blood, leading to jaundice. Blood samples were taken from three young men at one-week intervals, and the concentration of bilirubin in the serum was measured. The measured concentrations are shown in the following Table.

\begin{array}{ccc} Individual A & Individual B & Individual C \\ 14 & 20 & 32 \\ 20 & 27 & 41 \\ 23 & 32 & 41 \\ 27 & 34 & 55 \\ 27 & 34 & 55 \end{array}

A=c(14,20,23,27,27);  
B=c(20,27,32,34,34);  
C=c(32,41,41,55,55);  
cbind(mean(A), mean(B),mean(C));  
##      [,1] [,2] [,3]  
## [1,] 22.2 29.4 44.8  
cbind(var(A), var(B), var(C));  
##      [,1] [,2] [,3]  
## [1,] 29.7 35.8 100.2

Is there sufficient evidence to indicate a difference in mean bilirubin concentration for the
three young men? Fit appropriate linear model(s) to the data and test at the 0.05
level of significance. Provide conclusion in the context of this problem.

Complete model:

y_{i} = μ_{1} x_{i_{1}} + μ_{2} x_{i_{2}} + μ_{3} x_{i_{3}} + ε_{i}

where

x_{i j} = {\begin{cases} 1 & if i th meausurement belongs to j t h person \\ 0 & otherwise \end{cases}

$ε_{i} \overset{i i d}{\sim} N (0, σ^{2})$

[\begin{matrix} y_{1} \\ ⋮ \\ y_{5} \\ y_{6} \\ ⋮ \\ y_{10} \\ y_{11} \\ ⋮ \\ y_{15} \end{matrix}] = [\begin{matrix} 1 & 0 & 0 \\ ⋮ & ⋮ & ⋮ \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ ⋮ & ⋮ & ⋮ \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ ⋮ & ⋮ & ⋮ \\ 0 & 0 & 1 \end{matrix}] \cdot [\begin{matrix} μ_{1} \\ μ_{2} \\ μ_{3} \end{matrix}] + \vec{ε}

${\hat{β}}_{c} = [X_{c}^{T} X_{c}]^{- 1} X_{c}^{T} Y$

$X_{c}^{T} X_{c} = (\begin{matrix} 5 & 0 & 0 \\ 0 & 5 & 0 \\ 0 & 0 & 5 \end{matrix}) = (\begin{matrix} n & 0 & 0 \\ 0 & n & 0 \\ 0 & 0 & n \end{matrix})$

$X_{c}^{T} Y = [\begin{matrix} \sum_{1}^{5} y_{i} \\ \sum_{6}^{10} y_{i} \\ \sum_{11}^{15} y_{i} \end{matrix}] = [\begin{matrix} \sum_{}^{n_{1}} y_{i} \\ \sum_{}^{n_{1} + n_{2}} y_{i} \\ \sum_{}^{n_{1} + n_{2} + n_{3}} y_{i} \end{matrix}]$

\begin{aligned} ⟹ [X_{c}^{T} X_{c}]^{- 1} X_{c} Y = [\begin{array}{c} {\bar{y}}_{1} \\ {\bar{y}}_{2} \\ {\bar{y}}_{3} \end{array}] \\ ⟹ {\hat{Y}}_{c} = X_{c} {\hat{β}}_{c} = [\begin{array}{c} {\bar{Y}}_{1} \\ ⋮ \\ {\bar{Y}}_{1} \\ {\bar{Y}}_{2} \\ ⋮ \\ {\bar{Y}}_{2} \\ {\bar{Y}}_{3} \\ ⋮ \\ {\bar{Y}}_{3} \end{array}] \begin{matrix} \begin{array}{c}  \end{array}} n_{1} (5) times \\ \begin{array}{c}  \end{array}} n_{2} (5) times \\ \begin{array}{c}  \end{array}} n_{3} (5) times \end{matrix} \end{aligned}

\begin{aligned} SSE (c) & = \sum_{i = 1}^{N} {(Y_{i} - {\hat{Y}}_{i})}^{2} \\ = \sum_{i = 1}^{n_{1}} {(Y_{i} - {\bar{Y}}_{1})}^{2} + \sum_{i = n_{1} + 1}^{n_{1} + n_{2}} {(Y_{i} - {\bar{Y}}_{2})}^{2} + \sum_{i = n_{1} + n_{2} + 1}^{n_{1} + n_{2} + n_{3}} {(Y_{i} - {\bar{Y}}_{3})}^{2} \\ = (n_{1} - 1) s_{1}^{2} + (n_{2} - 1) s_{2}^{2} + (n_{3} - 1) s_{3}^{2} \end{aligned}

Reduced Model:

y_{i} = μ + ε_{i}, ε_{i} \overset{i i d}{\sim} N (0, σ^{2})

[\begin{matrix} y_{1} \\ ⋮ \\ y_{15} \end{matrix}] = [\begin{matrix} 1 \\ ⋮ \\ 1 \end{matrix}] μ + \vec{ε}

$X_{R}^{T} X_{R} = 15 ⟹ [X_{R}^{T} X_{R}]^{- 1} = \frac{1}{15} = \frac{1}{N}$

\begin{aligned} X_{R}^{T} Y & = \sum_{}^{15} y_{i} ⟹ {\hat{β}}_{R} = \frac{1}{N} \sum_{}^{N} y_{i} \\ {\hat{Y}}_{R} & = X_{R} {\hat{β}}_{R} = [\begin{array}{c} \bar{y} \\ ⋮ \\ \bar{y} \end{array}] \end{aligned}

SSE (R) = \sum_{i = 1}^{N} {(Y_{i} - {\hat{Y}}_{i})}^{2} = \sum_{i = 1}^{N} {(Y_{i} - \bar{Y})}^{2} = (N - 1) \cdot S_{DATASET}^{2}

Hypothesis Test

$H_{0} : μ_{1} = μ_{2} = μ_{3} = μ$

F_{*} = \frac{\frac{(S S E (R) - S S E (C)}{(# of C betas - # of R betas))}}{\frac{S S E (C)}{N - # of C betas}}

\begin{aligned} S S E (C) & = 4 [29.7 + 35.8 + 100.2] = 662.8 \\ S S E (R) & = (N - 1) s_{D A T A S E T}^{2} = 1995.733 \end{aligned}

⟹ F_{*} = \frac{(\frac{1995.733 - 662.8}{3 - 1})}{(\frac{662.8}{15 - 3})} \approx 12.0664

R R = {F_{*} > F_{2, 12, α = 0.05} = 3.89}

our $F_{*} \in R R$ so reject $H_{0}$

Example 2 Testing Means

Rhree brands of batteries are under study. It is suspected that the average life (in
weeks) of the three brands is different. Five batteries of each brand are tested with the
following results.

\begin{array}{ccc} Brand 1 & Brand 2 & Brand 3 \\ One & Two & Three \\ 100 & 76 & 108 \\ 96 & 80 & 100 \\ 92 & 75 & 96 \\ 96 & 84 & 98 \\ 92 & 82 & 100 \end{array}

                 One Two Three  
Sample means     95.2 79.4 100.4  
                 One Two Three  
Sample variances 11.2 14.8 20.8

Answer each of the following questions by fitting an appropriate multiple linear regression
model and assuming that the errors are independent and Normally distributed with mean
0 and variance σ2. You have to show all your work to get full credit. Simplify final answers and round to 4 decimal places where appropriate.

i) Find $\hat{β}$ , the vector of least-squares estimators for your model. Provide its expression
and numerical value.

\begin{aligned} \hat{β} & = (X^{T} X)^{- 1} X^{T} Y = (\begin{array}{c} \frac{1}{5} & 0 & 0 \\ 0 & \frac{1}{5} & 0 \\ 0 & 0 & \frac{1}{5} \end{array}) (\begin{array}{c} \sum_{}^{5} y_{i} \\ \sum_{}^{10} y_{i} \\ \sum_{}^{15} y_{i} \end{array}) = (\begin{array}{c} {\bar{y}}_{1} \\ {\bar{y}}_{2} \\ {\bar{y}}_{3} \end{array}) \\ = (\begin{array}{c} 95.2 \\ 79.4 \\ 100.4 \end{array}) \end{aligned}

\begin{aligned} s^{2} & = \frac{S S E}{N - #betas} = \frac{S S E}{15 - 3} \\ \sum_{}^{} (y_{i} - {\hat{y}}_{i})^{2} = \sum_{1}^{5} [y_{i} - {\bar{y}}_{1}]^{2} + \sum_{6}^{10} [y_{i} - {\bar{y}}_{2}]^{2} + \sum_{11}^{15} (y_{i} - \bar{y})^{2} \end{aligned}

Target: $μ_{2} = [0, 1, 0] [\begin{matrix} μ_{1} \\ μ_{2} \\ μ_{3} \end{matrix}] = a^{T} β$
For these types of questions $a$ is a central components

$E [a^{T} \hat{β}] = a^{T} E [\hat{β}] = a^{T} β$
$c o v [a^{T} \hat{β}] = a^{T} c o v (\hat{β}) a = σ^{2} a^{T} (X^{T} X)^{- 1} a$

a^{T} (X^{T} X)^{- 1} a = [\begin{matrix} 0 & 1 & 0 \end{matrix}] (\begin{matrix} \frac{1}{5} & 0 & 0 \\ 0 & \frac{1}{5} & 0 \\ 0 & 0 & \frac{1}{5} \end{matrix}) [\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}] = \frac{1}{5}

\frac{a^{T} \hat{β} - a^{T} β}{σ \sqrt{a^{T} (X^{T} X)^{- 1} a}} \sim N (0, 1)

\frac{[N - 3] s^{2}}{σ^{2}} \sim χ_{N - 3}^{2}

\frac{a^{T} \hat{β} - a^{T} β}{s \sqrt{a^{T} (X^{T} X)^{- 1} a}} \sim t (N - 3)

CI $= a^{T} \hat{β} \pm t_{\frac{α}{2}, N - 3} s \sqrt{a^{T} (X^{T} X)^{- 1} a}$

For $μ_{3} - μ_{2} = [0, - 1, 1] [\begin{matrix} μ_{1} \\ μ_{2} \\ μ_{3} \end{matrix}]$
steps are the same but $a^{T} = [0, - 1, 1]$
Estimator ${\bar{y}}_{3} - {\bar{y}}_{2} = a^{T} \hat{β}$

$a^{T} (X^{T} X)^{- 1} a = \frac{1}{n_{2}} + \frac{1}{n_{3}}$

General Testing of Parameters

t_{*} = \frac{a^{T} \hat{β} - (a^{T} β)_{0}}{s \sqrt{a^{T} (X^{T} X)^{- 1} a}}

$\sqrt{s^{2} a^{T} (X^{T} X)^{- 1} a} = S E (a^{T} \hat{β})$

Example Individual Variables

In a small-scale study of the relation between degree of brand liking (Y ) and moisture content (X1) and sweetness (X2) of the product, the following results were obtained (data are coded):

X1 X2 Y  
1 4 2 64  
2 4 4 73  
3 4 2 61  
4 4 4 76  
5 6 2 72  
6 6 4 80  
7 6 2 71  
8 6 4 83  
9 8 2 83  
10 8 4 89  
11 8 2 86  
12 8 4 93  
13 10 2 88  
14 10 4 95  
15 10 2 94  
16 10 4 100

A first-order model for mean brand liking, $E (Y)$ , as a function of moisture content $X_{1}$ and sweetness $X_{2}$ was fit to the data using R, i.e., $Y = β_{0} + β_{1} X_{1} + β_{2} X_{2} + ε$ .

modA=lm(y~x1 + x2);  
summary(modA);

             Estimate Std. Error t value  
(Intercept) 37.650 2.9961032 12.566323  
x1          4.425 0.3011197 14.695153  
x2          4.375 0.6733241 6.497614

modA=lm(y~x1 + x2);  
anova(modA);

	      Df Sum Sq Mean Sq F value  
x1        1 1566.45 1566.45 215.947  
x2        1 306.25 306.25 42.219  
Residuals 13 94.30 7.25

i) Is moisture content $X_{1}$ a statistically useful predictor of brand liking? Test using α = 0.01. Please, provide: hypotheses, test statistic, P-value or Rejection Region, and conclusion. Justify your answers.

$H_{0} : β_{1} = 0$ vs $H_{a} : β_{1} \neq 0$
Test stat $t_{*} \approx 14.6951$

R R = {t_{*} such that | t_{*} | > t_{\frac{α}{2}, 13}}

$t_{*} \in R ⟹$ reject $H_{0}$