<s>[INST] How to show whether a $3\times 4$ matrix has no solution, a unique solution or infinitely many solutions? The system is :
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
-2 & 5 & -4 & -1 & | & b \\
1 & -10 & 22 & 8 & | & c
\end{matrix}
$$
After Gaussian elimination, I found that
$$
\begin{array}{cccc|cc}
1 & -4 & 6 & a &  & 0 \\
0 & 1 & -\tfrac{8}{3} & - \left( 2a- \tfrac{1}{3} \right) & & - \tfrac{1}{3}b \\
0 & 0 & 0 & 10-5a & & c-2b
\end{array}
$$
Is it correct and I can continue to determine whether there is no solution, a unique solution or infinitely many solutions?
Here are the operations:
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
-2 & 5 & -4 & -1 & | & b \\
1 & -10 & 22 & 8 & | & c
\end{matrix}
$$
$R_2+2R_1\rightarrow R_2$
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
0 & -3 & 8 & 2a-1 & | & b \\
1 & -10 & 22 & 8 & | & c
\end{matrix}
$$
$R_3-R_1\rightarrow R_3$
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
0 & -3 & 8 & 2a-1 & | & b \\
0 & -6 & 16 & 8-a & | & c
\end{matrix}
$$
$R_3-2R_2\rightarrow R_3$
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
0 & -3 & 8 & 2a-1 & | & b \\
0 & 0 & 0 & 10-5a & | & c-2b
\end{matrix}
$$
$-\frac 13(R_2)\rightarrow R_2$
$$
\begin{matrix}
1 & -4 & 6 & a & | & 0 \\
0 & 1 & -\frac 83 & -\tfrac{2a-1}{3} & | & -\frac 13b \\
0 & 0 & 0 & 10-5a & | & c-2b
\end{matrix}
$$
 [/INST] Constraints on $a$
In general, we have existence of a solution when the data vector is in the column space of the target matrix $\mathbf{A}$. That is, if the data vector can be written as a linear combination of the fundamental columns of $\mathbf{A}$.
Additionally, if the nullspace $\mathcal{N} \left( \mathbf{A}^{*} \right)$ is trivial, then the solution is also unique.
Look for the fundamental columns. Start with the reduced row echelon form:
$$
\begin{align}
  \mathbf{A} &\mapsto \mathbf{E}_{\mathbf{A}} \\
%
\left[
\begin{array}{rrrr}
 1 & -4 & 6 & a \\
 -2 & 5 & -4 & -1 \\
 1 & -10 & 22 & 8 \\
\end{array}
\right]
%
&\mapsto 
%
\left[
\begin{array}{rrrr}
 \color{blue}{1} & 0 & -\frac{14}{3} & 0 \\
 0 & \color{blue}{1} & -\frac{8}{3} & 0 \\
 0 & 0 & 0 & \color{blue}{1} \\
\end{array}
\right]
%
\end{align}
$$
The fundamental columns are marked by blue pivots. Because the 3rd column is linearly dependent, we can ignore it and study a simpler, equivalent form:
$$
  \hat{\mathbf{A}} =
%
\left[
\begin{array}{rrrr}
  1 &  -4 &  a \\
 -2 &   5 & -1 \\
  1 & -10 &  8 \\
\end{array}
\right]
$$
The two matrices have the same range space:
$$
  \mathcal{R} \left(       \mathbf{A} \right) = 
  \mathcal{R} \left( \hat{ \mathbf{A} } \right)
$$
The determinant will determine when there is a nullspace.
$$
 \det \hat{\mathbf{A}} = 15 \left( a - 2 \right)
$$
When $a=2$ there are no longer three linearly independent columns. If a solution exists, it will not be unique.
Constraints on $b$, $c$
If $b=c=0$, we are out of the range space and probing the null space. When
$a\ne2$ the nullspace is
$$
\mathcal{N} \left( \mathbf{A}^{*} \right) = \text{span }
%
\left\{ \,
\left[ \begin{array}{c}
14 \\ 8 \\ 3 \\ 0
\end{array} \right]
\, \right\}
$$
When
$a=2$ the dimension of the nullspace increases
$$
\mathcal{N} \left( \mathbf{A}^{*} \right) = \text{span }
%
\left\{ \,
\left[ \begin{array}{c}
14 \\ 8 \\ 3 \\ 0
\end{array} \right]
,\,
\left[ \begin{array}{c}
2 \\ 1 \\ 0 \\ 1
\end{array} \right]
\, \right\}
$$
When $a\ne 2$ and either $b\ne0$ or $c\ne0$ we are guaranteed a unique solution because $\mathcal{R} \left( \mathbf{A} \right) = \mathbb{C}^{3}$.
</s>