<s>[INST] Product of nilpotent matrices. Let $A$ and $B$ be $n \times n$ complex matrices  and
let  $[A,B] = AB - BA$.
Question: If  $A , B$ and  $[A,B]$  are all nilpotent matrices,
is it necessarily true that  $\operatorname{trace}(AB) = 0$?
If,in fact, $[A,B] = 0$, then we can take $A$ and $B$ to be strictly upper triangular matrices so that the answer would be yes in this very special case.
 [/INST] No. While the conjecture is true for $n=2$ and computer experiments suggest that it may also be true for $n=3$, counterexamples are abundant when $n=4$. Let $A$ be the $4\times4$ Jordan block. The following $B$s are some computer generated random counterexamples that satisfy the condition $B^4=(AB-BA)^4=0$ but $\mathrm{trace}(AB)\neq0$.
\begin{align*}
\pmatrix{0&0&0&0\\ 0&1&1&0\\ -1&-1&-1&0\\ 1&0&0&0},
&\pmatrix{
 4& -4&  4& -6\\
 2& -2&  2& -3\\
14&-14& 14&-13\\
16&-16& 16&-16
},\\
\\
\pmatrix{
 1&  1&  1&  2\\
 0&  0&  0&  0\\
 1&  2&  1&  2\\
-1& -2& -1& -2\\
},
&\pmatrix{
-1& -1&  0&  0\\
 1&  1&  0&  0\\
 0&  0&  0&  1\\
 1&  1&  0&  0\\
}.
\end{align*}
</s>