# Linear Regression: Using a Decomposition (Cholesky Method)
--------------------------------

This script will use TensorFlow's function, `tf.cholesky()` to decompose our design matrix and solve for the parameter matrix from linear regression.

For linear regression we are given the system $A \cdot x = y$.  Here, $A$ is our design matrix, $x$ is our parameter matrix (of interest), and $y$ is our target matrix (dependent values).

For a Cholesky decomposition to work we assume that $A$ can be broken up into a product of a lower triangular matrix, $L$ and the transpose of the same matrix, $L^{T}$.

Note that this is when $A$ is square.  Of course, with an over determined system, $A$ is not square.  So we factor the product $A^{T} \cdot A$ instead.  We then assume:

$$A^{T} \cdot A = L^{T} \cdot L$$

For more information on the Cholesky decomposition and it's uses, see the following wikipedia link: [The Cholesky Decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition)

Given that $A$ has a unique Cholesky decomposition, we can write our linear regression system as the following:


\begin{align}
A \cdot x &= y \\
\Longleftrightarrow A^T \cdot A \cdot x &= A^T \cdot y \\
\Longleftrightarrow L^{T} \cdot L \cdot x &= A^{T} \cdot y
\end{align}

Then we break apart the system as follows:

$$L^{T} \cdot z = A^{T} \cdot y$$

and

$$L \cdot x = z$$

The steps we will take to solve for $x$ are the following

 1. Compute the Cholesky decomposition of $A$, where $A^{T} \cdot A = L^{T} \cdot L$.

 2. Solve ($L^{T} \cdot z = A^{T} \cdot y$) for $z$.
 
 3. Finally, solve ($L \cdot x = z$) for $x$.
 
We start by loading the necessary libraries.


```python
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
ops.reset_default_graph()
```

Next we create a graph session


```python
sess = tf.Session()
```

We use the same method of generating data as in the prior recipe for consistency.


```python
# Create the data
x_vals = np.linspace(0, 10, 100)
y_vals = x_vals + np.random.normal(0, 1, 100)
```

We generate the design matrix, $A$.


```python
# Create design matrix
x_vals_column = np.transpose(np.matrix(x_vals))
ones_column = np.transpose(np.matrix(np.repeat(1, 100)))
A = np.column_stack((ones_column, x_vals_column))
```

Next, we generate the 


```python
# Create y matrix
y = np.transpose(np.matrix(y_vals))

# Create tensors
A_tensor = tf.constant(A)
y_tensor = tf.constant(y)
```

Now we calculate the square of the matrix $A$ and the Cholesky decomposition.


```python
# Find Cholesky Decomposition
tA_A = tf.matmul(tf.transpose(A_tensor), A_tensor)
L = tf.cholesky(tA_A)  # tf.cholesky returns L only. L is Lower triangular matrix
```

We solve the first equation. (see step 2 in the intro paragraph above)


```python
# Solve L*z=t(A)*y
tA_y = tf.matmul(tf.transpose(A_tensor), y)
z = tf.matrix_solve(L, tA_y)
```

We finally solve for the parameter matrix by solving the second equation (see step 3 in the intro paragraph).


```python
# Solve L' * sol = z
sol = tf.matrix_solve(tf.transpose(L), z)

solution_eval = sess.run(sol)  # shape=(2, 1)
```

Extract the coefficients and create the best fit line.


```python
# Extract coefficients
slope = solution_eval[1][0]
y_intercept = solution_eval[0][0]

print('slope: ' + str(slope))
print('y_intercept: ' + str(y_intercept))

# Get best fit line
best_fit = []
for i in x_vals:
    best_fit.append(slope*i+y_intercept)
```

    slope: 0.9669761720326052
    y_intercept: 0.056840205248713006


Finally, we plot the fit with Matplotlib.


```python
# Plot the results
plt.style.use("ggplot")
plt.plot(x_vals, y_vals, 'o', label='Data')
plt.plot(x_vals, best_fit, 'b-', label='Best fit line', linewidth=3)
plt.legend(loc='upper left')
plt.show()
```


```python

```

534476f0e8fedd9becceab9e22d387111293ff41
ipynb
Jupyter Notebook
03_Linear_Regression/02_Implementing_a_Decomposition_Method/02_lin_reg_decomposition.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
2021-02-27T16:16:02.000Z
2021-02-27T16:16:02.000Z
03_Linear_Regression/02_Implementing_a_Decomposition_Method/02_lin_reg_decomposition.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
2018-03-07T14:31:22.000Z
2018-03-07T15:04:17.000Z
03_Linear_Regression/02_Implementing_a_Decomposition_Method/02_lin_reg_decomposition.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
Qwen/Qwen-72B
1. YES
2. YES