<blockquote>
    <h1>Exercise 3.11</h1>
    <p>
        In this problem we will investigate the $t$-statistic for the null hypothesis $H_0 : \beta = 0$ in simple linear regression without an intercept. To begin, we generate a predictor $x$ and a response $y$ as follows.<br>
        <code>> set.seed(1)
> x=rnorm(100)
> y=2*x*rnorm(100)</code></p>
    <ol>
        <li>Perform a simple linear regression of $y$ onto $x$, <i>without</i> an intercept. Report the coefficient estimate $\hat{\beta}$, the standard error of this coefficient estimate, and the $t$-statistic and $p$-value associated with the null hypothesis $H_0:\beta = 0$. Comment on these results. (You can perform regression without an intercept using the command <code>lm(y~x+0)</code>.)</li>
        <li>Now perform a simple linear regression of $x$ onto $y$ without an intercept, and report the coefficient estimate, its standard error, and the corresponding $t$-statistic and $p$-values associated with the null hypothesis $H_0:\beta = 0$. Comment on these results.</li>
        <li>What is the relationship between the results obtained in 1 and 2?</li>
        <li>For the regression of $Y$ onto $X$ without an intercept, the $t$-statistic for $H_0:\beta = 0$ takes the form $\hat{\beta} / \mathrm{SE}(\hat{\beta})$, where $\hat{\beta}$ is given by [equation] (3.38), and where
        $$
            \mathrm{SE}(\hat{\beta}) = \sqrt{\frac{\sum_{i=1}^n \left(y_i - x_i \hat{\beta}\right)^2}{(n-1)\sum_{i'=1}^n x_{i'}^2}} \,.
        $$
        (These formulas are slightly different from those given in Sections 3.1.1 and 3.1.2, since here we are performing regression without an intercept.) Show algebraically, and confirm numerically in R , that the $t$-statistic can be written as
        $$
            \frac{ (\sqrt{n-1}) \sum_{i=1}^n x_i y_i }{ \sqrt{ (\sum_{i=1}^n x_i^2) (\sum_{i'=1}^n y_{i'}^2) - (\sum_{i'=1}^n x_{i'} y_{i'})^2 } } \,.
        $$</li>
        <li>Using the results from 4, argue that the $t$-statistic for the regression of $y$ onto $x$ is the same as the $t$-statistic for the regression of $x$ onto $y$.</li>
        <li>In R, show that when regression is performed <i>with</i> an intercept, the $t$-statistic for $H_0:\beta_1 = 0$ is the same for the regression of $y$ onto $x$ as it is for the regression of $x$ onto $y$.</li>
    </ol>
</blockquote>


```python
import pandas as pd
import numpy as np
import numpy.testing as npt
import matplotlib.pyplot as plt

%run ../../customModules/usefulFunctions.ipynb
# https://stackoverflow.com/questions/34398054/ipython-notebook-cell-multiple-outputs
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

import statsmodels.api as sm
```


```python
np.random.seed(42)
x = np.random.normal(size=100, loc=0, scale=1)
y = 2 * x * np.random.normal(size=100, loc=0, scale=1)
df_x = pd.DataFrame({'x': x})
df_y = pd.DataFrame({'y': y})
fig = plt.figure(constrained_layout=True, figsize=(3, 3))
ax1 = fig.add_subplot(211)
_ = ax1.scatter(df_x, df_y)
_ = ax1.set_xlabel('x')
_ = ax1.set_ylabel('y')
ax2 = fig.add_subplot(212)
_ = ax2.scatter(df_y, df_x)
_ = ax2.set_xlabel('y')
_ = ax2.set_ylabel('x')

```

<h3>Exercise 3.11.1</h3>
<blockquote>
    <i>Perform a simple linear regression of $y$ onto $x$, <i>without</i> an intercept. Report the coefficient estimate $\hat{\beta}$, the standard error of this coefficient estimate, and the $t$-statistic and $p$-value associated with the null hypothesis $H_0:\beta = 0$. Comment on these results. (You can perform regression without an intercept using the command <code>lm(y~x+0)</code>.)</i>
</blockquote>



```python
model = sm.OLS(df_y, df_x)
fitted = model.fit()
coef_result1 = fitted.params[0] # store the value of the slope coefficient for later use
t_stat_result1 = fitted.tvalues[0]  # store the value t-statistic for later use
fitted.summary()
createSimpleLinearRegressionPlot(df_x, df_y, fitted_model=fitted)
```

<p>The estimation of the coefficient $\hat{\beta}$ is $0.69$, with an estimated standard error of $0.21$, a $t$-statistic of $3.23$ and a $p$ value of $0.2 \%$. This indicates that there is a significant relation between $x$ and $y$.</p>

<h3>Exercise 3.11.2</h3>
<blockquote>
    <i>Now perform a simple linear regression of $x$ onto $y$ without an intercept, and report the coefficient estimate, its standard error, and the corresponding $t$-statistic and $p$-values associated with the null hypothesis $H_0:\beta = 0$. Comment on these results.</i>
</blockquote>


```python
model = sm.OLS(df_x, df_y)
fitted = model.fit()
fitted.tvalues
t_stat_result2 = fitted.tvalues[0]  # store the value t-statistic for later use
fitted.summary()
createSimpleLinearRegressionPlot(df_y, df_x, fitted_model=fitted)
```

<p>The estimation of the coefficient $\hat{\beta}$ is $0.14$, with an estimated standard error of $0.04$, a $t$-statistic of $3.23$ and a $p$ value of $0.2 \%$. This again indicates that there is a significant relation between $x$ and $y$.</p>

<h3>Exercise 3.11.3</h3>
<blockquote>
    <i>What is the relationship between the results obtained in 1 and 2?</i>
</blockquote>

<p>For result 1 we have assumed
$$
y_i = \hat{\beta} * x_i + e_i  \,,
$$which can be rewritten as
$$
x_i = \frac{1}{\hat{\beta}} y_i-\frac{e_i}{\hat{\beta}} \,.
$$
We can easily compute this calculation as a sanity check.</p>


```python
error = y - coef_result1 * x
x2 = (y - error) / coef_result1
difference = x - x2 

# assert that the difference vector indeed approximates a zero vector
npt.assert_almost_equal(difference, np.zeros(difference.shape), decimal=10)
```

<h3>Exercise 3.11.4</h3>
<blockquote>
    <i>For the regression of $Y$ onto $X$ without an intercept, the $t$-statistic for $H_0:\beta = 0$ takes the form $\hat{\beta} / \mathrm{SE}(\hat{\beta})$, where $\hat{\beta}$ is given by [equation] (3.38), and where
        $$
            \mathrm{SE}(\hat{\beta}) = \sqrt{\frac{\sum_{i=1}^n \left(y_i - x_i \hat{\beta}\right)^2}{(n-1)\sum_{i'=1}^n x_{i'}^2}} \,.
        $$
        (These formulas are slightly different from those given in Sections 3.1.1 and 3.1.2, since here we are performing regression without an intercept.) Show algebraically, and confirm numerically in R , that the $t$-statistic can be written as
        $$
            \frac{ (\sqrt{n-1}) \sum_{i=1}^n x_i y_i }{ \sqrt{ (\sum_{i=1}^n x_i^2) (\sum_{i'=1}^n y_{i'}^2) - (\sum_{i'=1}^n x_{i'} y_{i'})^2 } } \,.
        $$</i>
</blockquote>

<p>Note that equation (3.38) is given by
$$
     \hat{\beta} = \frac{\sum_{i=1}^n x_i y_i}{\sum_{i'=1}^n x_{i'}^2} \,.
$$
Thus, we can write the $t$-statistic as
\begin{equation}
    \begin{aligned}
        t & =  \frac{\sum_{i=1}^n x_i y_i}{\sum_{i=1}^n x_{i}^2} \sqrt{\frac{(n-1)\sum_{i=1}^n x_{i}^2}{\sum_{i=1}^n \left(y_i - x_i \hat{\beta}\right)^2}} \\&
          = \frac{\sqrt{n-1} \sum_{i=1}^n x_i y_i}{ \sqrt{ \sum_{i=1}^n x_i^2 \sum_{i=1}^n \left(y_i - x_i \hat{\beta}\right)^2 } } \\&
          = \frac{\sqrt{n-1} \sum_{i=1}^n x_i y_i}{ \sqrt{ \sum_{i=1}^n x_i^2 \sum_{i=1}^n \left( y_i^2 - 2 \hat{\beta} x_i y_i + \hat{\beta} x_i^2 \right) } } \\&
          = \frac{\sqrt{n-1} \sum_{i=1}^n x_i y_i}{\sqrt{ \sum_{i=1}^n x_i^2 \sum_{i=1}^n y_i^2 - \hat{\beta} \sum_{i=1}^n x_i^2 \left( 2 \sum_{i=1}^n x_i y_i - \hat{\beta} \sum_{i=1}^n x_i^2 \right) }} \,.
    \end{aligned}
\end{equation}
Now, using the estimate for $\hat{\beta}$ given by equation (3.38) to further simply the expression for the $t$-statistic
\begin{equation}
    \begin{aligned}
        t & = \frac{\sqrt{n-1} \sum_{i=1}^n x_i y_i}{\sqrt{ \sum_{i=1}^n x_i^2 \sum_{i=1}^n y_i^2 - \sum_{i=1}^n x_i y_i \left( 2 \sum_{i=1}^n x_i y_i - \sum_{i=1}^n x_i y_i \right) }} \\&
          = \frac{ (\sqrt{n-1}) \sum_{i=1}^n x_i y_i }{ \sqrt{ (\sum_{i=1}^n x_i^2) (\sum_{i'=1}^n y_{i'}^2) - (\sum_{i'=1}^n x_{i'} y_{i'})^2 } } \,.
    \end{aligned}
\end{equation}
as required. We will now also confirm this equation using Python.</p>


```python
n = x.shape[0]
t = ( (n-1)**0.5 * np.sum(x*y) ) / ( np.sum(x**2) * np.sum(y**2) - (np.sum(x*y))**2  )**0.5
# assert that this result is equal to the t-statistics computer for the above two regressions
assert abs(t - t_stat_result1) <= 1e-10
assert abs(t - t_stat_result2) <= 1e-10
```

<h3>Exercise 3.11.5</h3>
<blockquote>
    <i>Using the results from 4, argue that the $t$-statistic for the regression of $y$ onto $x$ is the same as the $t$-statistic for the regression of $x$ onto $y$.</i>
</blockquote>

<p>We can clearly see from the formula of the $t$-statistic that it is a symmetric function of the $x_i$ and $y_i$ variables.</p>

<h3>Exercise 3.11.6</h3>
<blockquote>
    <i>In R, show that when regression is performed <i>with</i> an intercept, the $t$-statistic for $H_0:\beta_1 = 0$ is the same for the regression of $y$ onto $x$ as it is for the regression of $x$ onto $y$.</i>
</blockquote>



```python
df_x.insert(0, 'Intercept', 1)
df_y.insert(0, 'Intercept', 1)
```


```python
model = sm.OLS(df_y[['y']], df_x)
fitted = model.fit()
fitted.summary()
createSimpleLinearRegressionPlot(df_x, df_y[['y']], fitted_model=fitted)
```


```python
model = sm.OLS(df_x[['x']], df_y)
fitted = model.fit()
fitted.summary()
createLinearRegressionPlot(df_y, df_x[['x']], fitted_model=fitted)
```

<p>We see that $\beta_0$ is estimated to be very close to zero, which is expected because the data used to fit the model was generated without an intercept. This is also reflected in that both models (with and without an intercept) have estimated approximately the same value for $\beta_1$, and their $t$-statistics also have very similar values.</p>

5c5c97955f1676efb55220b2e90c40d65f349e7a
ipynb
Jupyter Notebook
Chapter03/Exercise11/3_11.ipynb
HerrBroekman/Applied-solution-of-An-Introduction-to-Statistical-Learning-written-in-Python
7c0e26f2ca13030472db7715ab33ef248dddc6ec
MIT
2020-02-26T17:30:27.000Z
2020-02-26T17:30:27.000Z
Chapter03/Exercise11/3_11.ipynb
HerrBroekman/Applied-solution-of-An-Introduction-to-Statistical-Learning-written-in-Python
7c0e26f2ca13030472db7715ab33ef248dddc6ec
MIT
Chapter03/Exercise11/3_11.ipynb
HerrBroekman/Applied-solution-of-An-Introduction-to-Statistical-Learning-written-in-Python
7c0e26f2ca13030472db7715ab33ef248dddc6ec
MIT
2020-02-26T17:30:30.000Z
2020-02-26T17:30:30.000Z
Qwen/Qwen-72B
1. YES
2. YES