<s>[INST] For what values of a will the system have a unique solution, and for which pair of values (a,b) will the system have more than one solution consider the following linear system 
$$x+2y+2z=1\tag{1}$$
$$x+ay+3z=3\tag{2}$$
$$x+11y+az=b\tag{3}$$
in matrix form
$$\pmatrix{1&2&2&1\cr1&a&3&3\cr1&11&a&b\cr}$$
For what values of a will the system have a unique solution, and for which pair of values (a,b) will the system have more than one solution
My attempt at solving it:
-1*line 1 + line 2, and,  -1*line 1 + line 3
$$\pmatrix{1&2&2&1\cr0&a-2&1&2\cr0&9
&a-2&b-1\cr}$$
1/(a-2) * line 
$$\pmatrix{1&2&2&1\cr0&a-2&1&2\cr0&9/(a-2)&1&(b-1)/(b-2)\cr}$$
-1*line 3 + line 2
$$\pmatrix{1&2&2&1\cr0&((a-5)(a+1))/(a-2)&0&(-b+2a-3)/(a-2)\cr0&9/(a-2)&1&(b-1)/(b-2)\cr}$$
I don't know how to proceed beyond this, help guide me stack exchange!
Also if someone could touch up my matrices formatting that would be soooo cool.
 [/INST] The original system is equivalent to the following under-determined system:
$$x+2y+2z+w=0\tag{1}$$
$$x+ay+3z+3w=0\tag{2}$$
$$x+11y+az+bw=0\tag{3}$$
Solve $z,w$ from (1) and (2) and substitution of it into (3) leads to:
$$(3-2a+b)x+(33+a^2+6b-2a(3+b))y=f(a,b)x+g(a,b)y=0\tag{4}$$
(1)If $a=-1,b=-5$ or $a=5,b=7$, then $f(a,b)=g(a,b)=0$. Thus $x,y$ can take any values.
(2) If $a\not=-1,5$ and $b=2a-3$, then $f(a,b)=0$ but $g(a,b)\not=0$, so $y=0$ and $x$ can take any value.
(3) If $a\not=-1,5$ and $b=\frac{33-6a+a^2}{2(a-3)}$, then  $g(a,b)=0$ but $f(a,b)\not=0$, so $x=0$ and $y$ can take any value.
(4) If $a\not=-1,5$, $b\not=2a-3,\frac{33-6a+a^2}{2(a-3)}$, then $f(a,b)\not =0,g(a,b)\not=0$, so $x=-\frac{g(a,b)}{f(a,b)}y$ and $y$ can take any value.
</s>