<s>[INST] Finding the best solution to an inconsistent system $A\mathbf{u} = \mathbf{b}$. Let $A = \begin{bmatrix}
-1 & 1 \\
2 & -1 \\
1 & 1
\end{bmatrix}$ and $\mathbf{b} = \begin{bmatrix}
1 \\
2 \\
0
\end{bmatrix}$.
1. Find a "best solution" to the inconsistent system $A\mathbf{u} = \mathbf{b}$.
2. Find the orthogonal projection of $\mathbf{b}$ onto the column space of $A$.  
For the second question the column space of $A$ has vectors that are all linearly independent. We first find the projection matrix given by $P = A(A^TA)^{-1}A^T$. Lets first calculate $A^TA$
\begin{equation*}
A^TA = \begin{bmatrix}
-1 & 2 & 1 \\
1 & -1 & 1
\end{bmatrix}
\begin{bmatrix}
-1 & 1 \\
2 & -1 \\
1 & 1
\end{bmatrix} = \begin{bmatrix}
6 & -2 \\
-2 & 3
\end{bmatrix}.
\end{equation*}
This means that
\begin{equation*}
(A^TA)^{-1} = \frac{1}{14}\begin{bmatrix}
3 & 2 \\
2 & 6
\end{bmatrix}.
\end{equation*}
Hence,
\begin{equation*}
\begin{split}
P = A(A^TA)^{-1}A^T &= \frac{1}{14}\begin{bmatrix}
-1 & 1 \\
2 & -1 \\
1 & 1
\end{bmatrix}
\begin{bmatrix}
3 & 2 \\
2 & 6
\end{bmatrix}
\begin{bmatrix}
-1 & 2 & 1 \\
1 & -1 & 1
\end{bmatrix} \\
&=  \frac{1}{14}\begin{bmatrix}
-1 & 1 \\
2 & -1 \\
1 & 1
\end{bmatrix}
\begin{bmatrix}
-1 & 4 & 5 \\
4 & -2 & 8
\end{bmatrix} \\
&= \frac{1}{14}\begin{bmatrix}
5 & -6 & 3 \\
-6 & 10 & 2 \\
3 & 2 & 13
\end{bmatrix}.
\end{split}
\end{equation*}
So the orthogonal projection of $\mathbf{b} = (1,2,0)$ onto the column space of $A$ is
\begin{equation*}
\frac{1}{14}\begin{bmatrix}
5 & -6 & 3 \\
-6 & 10 & 2 \\
3 & 2 & 13
\end{bmatrix}
\begin{bmatrix}
1 \\
2 \\
0 \\
\end{bmatrix} = \frac{1}{14}\begin{bmatrix}
-7 \\
14 \\
7
\end{bmatrix} = \begin{bmatrix}
-1/2 \\
1 \\
1/2
\end{bmatrix}.
\end{equation*}
Not sure what is meant by the "best solution" to the system. Any would help would be great!!!
 [/INST] As you said $A\mathbf{u} = \mathbf{b}$ is an inconsistent system that has no solution. So the best you can hope for is to find a $\mathbf{u}$ that makes $A\mathbf{u}$ "close to" $\mathbf{b}$. Or to say the same thing you can try to make $A\mathbf{u} - \mathbf{b}$ "as close to $\mathbf{0}$ as possible".
So the problem boils down to minimizing the size of $A\mathbf{u} - \mathbf{b}$ i.e. $||A\mathbf{u} - \mathbf{b}||$. Though in practice it is more convenient to minimize $F(\mathbf{u}) = ||A\mathbf{u} - \mathbf{b}||^2$ instead.
Now if you set $\mathbf{u} = \begin{bmatrix}x \\ y \end{bmatrix}$ you can multiply things out to get:
$$
A\mathbf{u} - \mathbf{b} = \begin{bmatrix} -x + y - 1 \\ 2x - y - 2 \\ x + y  \end{bmatrix}
$$ So that
$$
F(x, y) = ||A\mathbf{u} - \mathbf{b}||^2 = (-x + y -1)^2 + (2x - y - 2)^2 + (x + y)^2
$$
A standard way to try to minimize a multivariable function like $F(x, y)$ is to take the partial derivatives w.r.t. to $x$ and $y$ and set the resulting equations to $0$:
$$
\partial_x F(x, y) = 12x - 4y - 6 = 0
$$
$$
\partial_y F(x, y) = -4x + 6y + 2 = 0
$$
And solving these gives you $x = \frac{1}{2}$ and $y = 0$. If you want to convince yourself that these values of $x$ and $y$ do in fact minimize $F(x, y) = ||A\mathbf{u} - \mathbf{b}||^2$, you can use the Second Partial Derivative Test:
$$
\partial_{xx} F(x, y) = 12
$$
$$
\partial_{xy} F(x, y) = -4 = \partial_{yx} F(x, y)
$$
$$
\partial_{yy} F(x, y) = 6
$$ And note that the matrix of these second partials i.e. the Hessian of $F(x, y)$
$$
\begin{bmatrix} \partial_{xx} F(x, y) &\partial_{xy} F(x, y) \\
\partial_{yx} F(x, y) &\partial_{yy} F(x, y) \end{bmatrix} = \begin{bmatrix} 12 &-4 \\
-4 &6 \end{bmatrix}
$$ has positive determinant $56 > 0$ and furthermore, at $(\frac{1}{2}, 0)$ the value of $\partial_{xx} F(x, y)$ is always positive: $12 > 0$. So the Second Partial Derivative Test says that $(\frac{1}{2}, 0)$ does indeed minimize $F(x, y) = ||A\mathbf{u} - \mathbf{b}||^2$ and hence
$$
\mathbf{u} = \begin{bmatrix}\frac{1}{2} \\ 0 \end{bmatrix}
$$ is the best solution to our linear system.
</s>