<s>[INST] Find the least squares solution of the linear system...... Hi I'm looking for the least squares solutions of....
$$ \begin{pmatrix} 1&-1 \\ -1&2 \\ -1&0 \end{pmatrix} \begin{pmatrix} x\\y \end{pmatrix} = \begin{pmatrix} 3 \\ 4 \\ 5\end{pmatrix} $$
So Assuming this goes by $ A\vec{x}=\vec{b}$
Then using.... $A^T A \vec{x} = A^T \vec{b} $
$$ \begin{pmatrix}  1&-1&-1 \\ -1&2&0 \end{pmatrix}  \begin{pmatrix} 1&-1 \\ -1&2 \\ -1&0 \end{pmatrix} \vec{x}= \begin{pmatrix}  1&-1&-1 \\ -1&2&0 \end{pmatrix} \begin{pmatrix} 3 \\ 4 \\ 5\end{pmatrix}  $$
Which solves to be 
$$ \begin{pmatrix} 3&-3 \\ -3&5 \end{pmatrix} \vec{x} = \begin{pmatrix} -6 \\ 5 \end{pmatrix} $$
which i solve to get 
$$ \vec{x} = \begin{pmatrix} -7/3 \\ 3/2 \end{pmatrix}  $$
 [/INST] The normal equations are correct:
$$
\mathbf{A}^{*}\mathbf{A} x = \mathbf{A}^{*}b.
$$
The error is in the solution which should be
$$
 x = \left( \mathbf{A}^{*}\mathbf{A} \right)^{-1} \mathbf{A}^{*}b.
$$
The inverse is
$$
\left( \mathbf{A}^{*}\mathbf{A} \right)^{-1} = \frac{1} {6}
\left(
\begin{array}{cc}
 5 & 3 \\
 3 & 3 \\
\end{array}
\right)
$$
which provides
$$
 x = -\frac{1}{2}
\left(
\begin{array}{r}
  5 \\
  1
\end{array}
\right).
$$
</s>