<s>[INST] linear algebra - equations 
Determine real number(s)for  $a,b$ such that the system has no solution, has a unique solution, and has more than one solution: 
  $$\begin{align*}
x-2y+az-t&=1\\
-x+y-z+t&=-1\\
(a+1)y-a^2z+at&=0\\
(b+1)y-abz-a^2t&=b
\end{align*}$$

I could not transform the matrix into row reduced form.So that I could not find $a$ and $b$. I will be glad if someone could solve it.
 [/INST] Your matrix will have $a$ and $b$ indicated. You then have to be careful with them. 
(I hope I didn't make any silly arithmetic or algebra mistakes; but even if I did, you should be able to see how one proceeds: doesn't matter what $a$ and $b$ are, so long as you are careful not to divide by an expression containing them, since you won't know ahead of time that what you are dividing by is not zero).
The augmented matrix for the system is:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
-1 & 1 & -1 & 1 & -1\\
0 & a+1 & -a^2 & a & 0\\
0 & b+1 & -ab & -a^2 & b
\end{array}\right).$$
Proceed as usual. First, we add the first row to the second row:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & -1 & a-1 & 0 & 0\\
0 & a+1 & -a^2 & a & 0\\
0 & b+1 & -ab & -a^2 & b
\end{array}\right).$$
Then, multiply the second row by $-1$:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & 1 & 1-a & 0 & 0\\
0 & a+1 & -a^2 & a & 0\\
0 & b+1 & -ab & -a^2 & b
\end{array}\right).$$
Now, subtract $(a+1)$-times the second row from the third row; multiplying the second row by $-(a+1)$ gives $(0\quad -(a+1)\quad (a^2-1)\quad 0\>|\>0)$, so we have:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & 1 & 1-a & 0 & 0\\
0 & 0 & -1 & a & 0\\
0 & b+1 & -ab & -a^2 & b
\end{array}\right).$$
Multiply the third row by $-1$:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & 1 & 1-a & 0 & 0\\
0 & 0 & 1 & -a & 0\\
0 & b+1 & -ab & -a^2 & b
\end{array}\right).$$
Now subtract $(b+1)$-times the second row from the fourth row. Multiplying the second row by $-(b+1)$ gives $(0\quad (b+1)\quad (ab+a-b-1)\quad 0\>|\>0)$, so we get:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & 1 & 1-a & 0 & 0\\
0 & 0 & 1 & -a & 0\\
0 & 0 & a-b-1 & -a^2 & b
\end{array}\right).$$
Now, subtract $(a-b-1)$-times the third row from the fourth row. Multiplying the third row by $-(a-b-1)$ gives $(0\quad 0\quad (1+b-a)\quad (a^2-ab-a)\>|\>0)$, so we get:
$$\left(\begin{array}{rrrr|r}
1 & -2 & a & -1 & 1\\
0 & 1 & 1-a & 0 & 0\\
0 & 0 & 1 & -a & 0\\
0 & 0 & 0 & -(ab+a) & b
\end{array}\right).$$
Now we can figure out the rank easily enough.
If $-(ab+a) = -a(b+1)\neq 0$, then the coefficient matrix has rank $4$ and the augmented matrix has rank $4$, so the system has exactly one solution.
If, however, $-a(b+1)=0$ and $b\neq 0$, then the coefficient matrix would have rank $3$ and the augmented matrix would have rank $4$, so that would be no solutions.
And, finally, if $-a(b+1)=0$ and $b=0$, then both the coefficient and augmented matrices have rank $3$, so the system would have an infinite number of solutions.
So now, it's just a matter of determining what are the conditions on $a$ and $b$ that give you those three possibilities. Suggestion: Deal with them in the inverse order I listed them: first the case where $-a(b+1)=0$ and $b=0$, because you'll be able to say exactly what $a$ and $b$ are; then consider the middle case; and finally the first mentioned case.
</s>