<s>[INST] Way to verify a least-squares solution without actually solving for $x$ and $y$? I just found the least squares solution of the system $\mathbf{x}A = \mathbf{b} = \begin{pmatrix} x & y \end{pmatrix}\begin{pmatrix} 3 & 2 & 1 \\ 2 & 3 & 2\end{pmatrix} = \begin{pmatrix} 3 & 0 & 1\end{pmatrix}$ to be $\begin{pmatrix} x \\ y\end{pmatrix} = \begin{pmatrix} \frac{29}{21} \\ -\frac{2}{3}\end{pmatrix}$.  I now am tasked with  answering how this solution can be verified without solving for $x$ and $y$.
The system can be rewritten as follows: $\begin{pmatrix}3 & 2 \\ 2 & 3 \\ 1 & 2 \end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 3 \\ 0 \\ 1 \end{pmatrix}$.
Since the least-squares solution is the vector $\mathbf{x}$ that makes $\Vert b - A\mathbf{x} \Vert_{2}$ a minimum, should I try graphing the equations $3x + 2y = 3$, $2x + 3y = 0$, and $x + 2y = 1$, then pointing out on the graph that $\begin{pmatrix}x \\ y \end{pmatrix} = \begin{pmatrix}\frac{29}{21} \\ -\frac{2}{3} \end{pmatrix}$ is close to the point where the three lines intersect?
I figure there must be a more formal way of verifying the solution, but it is escaping me. Could somebody please let me know how I should do it?
 [/INST] By construction, the least squares solution minimizes the sum of the squares of the residuals
$$
 r^{2}(x,y) = \lVert A x - b \rVert_{2}^{2}.
$$
that is, the least squares solution can be defined as 
$$
  \left[
    \begin{array}{c}
      x \\ y
    \end{array}
  \right]_{LS}
  =
  \left\{
  \left[
    \begin{array}{c}
      x \\ y
    \end{array}
  \right]
    \in \mathbb{R}^{2} \colon \lVert A x - b \rVert_{2}^{2} \text{ is minimized}
  \right\}
$$
Show the proposed solution is a minimum by establishing that $r^{2}(x,y)_{LS}$ is the smallest value in a $\delta-$neighborhood of $x_{LS}$.
Let 
$$
  \left[
    \begin{array}{c}
      x \\ y
    \end{array}
  \right]
     = 
  \left[
    \begin{array}{c}
      x \\ y
    \end{array}
  \right]_{LS}
    +
  \left[
    \begin{array}{c}
      \delta_{1} \\ \delta_{2}
    \end{array}
  \right]
    =
  \left[
    \begin{array}{r}
      \frac{29}{21} + \delta_{1} \\ -\frac{2}{3} + \delta_{2}
    \end{array}
  \right]
.
$$
The norm of the residual error vector is 
$$
  \lVert A (x_{LS} + \delta) - b \rVert_{2}^{2} = \sqrt{\frac{32}{21} + 14 \delta_{1}^2 + 28 \delta_{1}\delta_{2} + 17\delta_{2}^2}.
$$
This establishes that the minimum value is attained when $\delta_{1} = \delta_{2} = 0$, that is when $x = x_{LS}$. Therefore, $x_{LS}$ is the solution of minimum norm.
</s>