<s>[INST] For what values of k does this system of equations have a unique / infinite / no solutions? My system of equations is:
\begin{cases}
x + 5y- 6z = 2 \\
kx + y - z = 3 \\
5x - ky + 3z = 7
\end{cases}
So the augmented matrix is:
$$ \left[
\begin{array}{ccc|c}
  1&5&-6&2\\
  k&1&-1&3\\
  5&-k&3&7
\end{array}
\right] $$
I reduced it to this (with the shown steps), but I'm not sure if i've done so correctly and i'm not sure if i need to do Gauss-Jordan elimination or just Gauss elimination:
\begin{align}
r_2 -> r_2 - 2r_1
\end{align}
\begin{align}
r_3 -> r+3 - 5r_1
\end{align}
\begin{align}
r_2 -> r_2 * \frac{1}{-9}
\end{align}
\begin{align}
r_1 -> r_1 - 5r_2
\end{align}
\begin{align}
r_3 -> r_3 +27r_2
\end{align}
$$ \left[
\begin{array}{ccc|c}
  1-5(\frac{k-2}{-9})&0&\frac{1}{9}&\frac{13}{9}\\
  \frac{k-2}{-9}&1&\frac{-11}{9}&\frac{1}{9}\\
  27(\frac{k-2}{-9})&-k+2&0&0
\end{array}
\right] $$
From this reduced from that i got, i noticed that there is an infinite solution for k=2.
However, I believe that i need to get one of the values in the 4th column of the matrix in terms of k so that i can find all the solutions i need, but i am unsure on how to do this.
Could someone please help me and show full working, including the steps taken to reduce the matrix?
 [/INST] From my comments above, I always find the determinant first which gives us
$$\det \begin{bmatrix}  1&5&-6 \\  k&1&-1 \\  5&-k&3 \end{bmatrix} = 2 (k-2) (3 k-2)$$
This tells us we may have to account for $$k = 2, k = \dfrac{2}{3}$$
The RREF is given by the steps


*

*Swap $R_1$ and $R_2$

*Set $R_2 \longleftarrow R_2 - \dfrac{1}{k} R_1$

*Set $R_3 \longleftarrow R_3 - \dfrac{5}{k} R_1$

*Swap $R_2$ and $R_3$

*Set $R_3 \longleftarrow R_3 - \dfrac{5k-1}{-k^2-5} R_2$

*Set $R_3 \longleftarrow - \dfrac{k^2+5}{2(3k^2-8k+4)} R_3$

*Set $R_2 \longleftarrow R_2 - \dfrac{3k+5}{k} R_3$

*Set $R_1 \longleftarrow R_1 + R_3$

*Set $R_2 \longleftarrow \dfrac{k}{-k^2-5} R_2$

*Set $R_1 \longleftarrow R_1 - R_2$

*Set $R_1 \longleftarrow \dfrac{1}{k} R_1$
This results in the RREF of
$$\left[
\begin{array}{ccc|c}
  1 & 0 & 0 & \dfrac{8}{3 k-2} \\
 0 & 1 & 0 & -\dfrac{24}{3 k-2} \\
 0 & 0 & 1 & \dfrac{-k-18}{3 k-2} \\
\end{array}
\right]$$
We can see that we have an issue for $k = \dfrac{2}{3}$.
Also, from the determinant, when $k = 2$, we have a RREF of 
$$\left[
\begin{array}{ccc|c}
 1 & 0 & \dfrac{1}{9} & \dfrac{13}{9} \\
 0 & 1 & -\dfrac{11}{9} & \dfrac{1}{9} \\
 0 & 0 & 0 & 0 \\
\end{array}
\right]$$
</s>