<s>[INST] When does a system of equations have no solution? I have performed Gaussian elimination on this matrix to reduce it to 
$$
\left[
\begin{array}{@{}ccc|c@{}}
-3&-1&2 & 1 \\
0& \frac{-5}{3}& \frac{10}{3} & \frac{8}{3} \\
0&0& a+2 & b + \frac{6}{5} \\
\end{array}
\right]
$$
I thought that setting $a$ equal to $-2$ and having $b$ not equal to $-\frac{6}{5}$ would be the answer to this problem, but it apparently isn't. Could someone please explain why?
 [/INST] Given a system of linear equations represented by the matrix equation: $\mathbf{A}\vec{x}=\vec{b}$, there is no unique set of solutions for $\det{\mathbf{A}}=0$.
Therefore, in your case:
$$\begin{bmatrix}-3 & -1 & 2 \\ 0 & -\frac{5}{3} & \frac{10}{3} \\ 0 & 0 & a+2\end{bmatrix}\begin{bmatrix}x \\ y \\ z\end{bmatrix}=\begin{bmatrix}1 \\ \frac{8}{3} \\ b+\frac{6}{5}\end{bmatrix}$$
So we are interested in the case when: 
$$\begin{vmatrix}-3 & -1 & 2 \\ 0 & -\frac{5}{3} & \frac{10}{3} \\ 0 & 0 & a+2\end{vmatrix}=0 \implies (5a+10)=0\implies a =-\frac{10}{5}=-2$$
</s>