<s>[INST] finding the values of a such that the system $AB\cdot x=0$ has infinitely many solutions the following matrix are given:
$$A=
    \begin{bmatrix}
    a & 1 & 1 \\
    a & a & 2 \\
    a & a & a \\
    \end{bmatrix}
$$
$$B=
    \begin{bmatrix}
    a-3 & 1 & 1 \\
    a-3 & a-3 & 2 \\
    a-3 & a-3 & a-3 \\
    \end{bmatrix}
$$
find the values of a such that the system $AB\cdot x=0$ has infinitely many solutions.
 $$$$I know that a matix has infinitely many solutions $\iff det(AB)=0$, so this is what I did :
$$A'=
    \begin{bmatrix}
    a & 1 & 1 \\
    0 & a-1 & 2 \\
    0 & 0 & a-2 \\
    \end{bmatrix}
$$
and $det(A) =0$ if $a\cdot(a-1)\cdot(a-2)=0$ , so if $a=0,1,2.$ A has infinitely many solutions.
$$B=
    \begin{bmatrix}
    a-3 & 1 & 1 \\
    0 & a-4 & 2 \\
    0 & 0 & a-5 \\
    \end{bmatrix}
$$
and $det(B) =0$ if $(a-3)\cdot(a-4)\cdot(a-5)=0$ , so if $a=3,4,5.$ B has infinitely many solutions. $$$$So $AB\cdot x=0$ , if $a=0,1,2,3,4,5$, BUT the textbook answer is $a=1,2,3$. What am I doing wrong? 
 [/INST] Your answer is right (I checked it) and therefore your textbook is wrong.
</s>