<s>[INST] When are matrices $AB$ and $BA$ similar? If $A$ or $B$ is invertible, it is easy to see that $AB$ and $BA$ are similar. I'm curious about when (maybe a strong sufficient or necessary condition) they become similar.
Here's what I have done:
There exists invertible matrices $P, Q$ such that $A=P\begin{pmatrix} I_r &\\ &0\end{pmatrix}Q$ where $r=\text{rank} A$. Let $B=Q^{-1}\begin{pmatrix} B_{11} & B_{12}\\ B_{21}&B_{22}\end{pmatrix}P^{-1}$.Then $AB=P\begin{pmatrix} B_{11} & B_{12}\\ 0&0\end{pmatrix}P^{-1}$ and $BA=Q^{-1}\begin{pmatrix} B_{11} & 0\\ B_{21}&0\end{pmatrix}Q$. Hence we just need to find when $\begin{pmatrix} B_{11} & B_{12}\\ 0&0\end{pmatrix}$ and $\begin{pmatrix} B_{11} & 0\\ B_{21}&0\end{pmatrix}$ are similar.
Now if $B_{11}$ is invertible, let $R=\begin{pmatrix} C  &0\\D&F\end{pmatrix}$ and then $R^{-1}\begin{pmatrix} B_{11} & 0\\ B_{21}&0\end{pmatrix}R=\begin{pmatrix} C^{-1}B_{11}C & 0\\ -F^{-1}DC^{-1}B_{11}C+F^{-1}B_{21}C&0\end{pmatrix}$. Known that $\begin{pmatrix} B_{11} & B_{12}\\ 0&0\end{pmatrix}$ is similar to $\begin{pmatrix} B_{11}^t & 0\\ B_{12}^t&0\end{pmatrix}$, we just check when the matrices above are equal.
First, there exists $C$ s.t. $C^{-1}B_{11}C=B_{11}^t$ since $B_{11}$ and $B_{11}^t$ are similar.
Second, $-F^{-1}DC^{-1}B_{11}C+F^{-1}B_{21}C=B_{12}^t\Leftrightarrow DB_{11}^t+FB_{12}^t=B_{21}C$. Hence if $\text{rank} \begin{pmatrix} B_{11} &B_{12} \end{pmatrix} = r$ then $D, F$ exist. Disappointingly, however, we need $F$ to be invertible. I cannot overcome this problem but I still have the romantic guess that $\text{rank}AB=\text{rank}BA=\text{rank}A$ even $\text{rank} AB=\text{rank} BA$ might be a sufficient condition (please tell me a counter-example so I could give up).
Of course, $F$ might not be invertible really. So if there is a way then that way must consider both $AB$ and $BA$. A weaker result we can get is if $B_{11}$ is invertible then $F=I, D=(B_{21}C-B_{12}^t){B_{11}^t}^{-1}$ satisfies the equation. But I cannot tell what $B_{11}$'s invertibility actually means. I wonder if there is a direct relation from $A$, $B$ (or $AB$, $BA$) to $B_{11}$.
I'll appreciate any discussions about this topic.
P.S. I know $AB$ and $BA$ are not similar when $A=\begin{pmatrix} 0 &1\\0&0\end{pmatrix}$ and $B=\begin{pmatrix} 1 &0\\0&0\end{pmatrix}$. That's just when $F$ is not invertible. Thanks for commenting anyway.
 [/INST] In general, two matrices $X$ and $Y$ are similar over an algebraically closed field $F$ if an only if $\operatorname{rank}\left((\lambda I-X)^k\right)=\operatorname{rank}\left((\lambda I-Y)^k\right)$ for all $\lambda\in F$ and $k\ge1$. However, in case $X=AB$ and $Y=BA$ for some square matrices $A$ and $B$, one only needs to check the previous condition for $\lambda=0$. In other words, $AB$ and $BA$ are similar if and only if $\operatorname{rank}\left((AB)^k\right)=\operatorname{rank}\left((BA)^k\right)$ for every integer $k\ge1$.
Merely $\operatorname{rank}(AB)=\operatorname{rank}(BA)$ is not enough to guarantee that $AB$ is similar to $BA$. Here is a random counterexample where $\operatorname{rank}(AB)=\operatorname{rank}(BA)=2$ but $(AB)^2=0\ne(BA)^2$:
\begin{aligned}
AB&=\pmatrix{1&1&0&0\\ 0&0&1&0\\ 0&0&1&0\\ 0&1&0&0}
\pmatrix{0&1&0&0\\ 0&0&1&0\\ 0&0&0&0\\ 0&1&1&1}
=\pmatrix{0&1&1&0\\ 0&0&0&0\\ 0&0&0&0\\ 0&0&1&0},\\
BA&=\pmatrix{0&1&0&0\\ 0&0&1&0\\ 0&0&0&0\\ 0&1&1&1}
\pmatrix{1&1&0&0\\ 0&0&1&0\\ 0&0&1&0\\ 0&1&0&0}
=\pmatrix{0&0&1&0\\ 0&0&1&0\\ 0&0&0&0\\ 0&1&2&0}.\\
\end{aligned}
However, that $\operatorname{rank}(AB)=\operatorname{rank}(BA)=\operatorname{rank}(A)$ is sufficient for the similarity between $AB$ and $BA$. In this case, we may continue from your approach. By Roth's removal rule, both $\pmatrix{B_{11}&B_{12}\\ 0&0}$ and $\pmatrix{B_{11}&0\\ B_{21}&0}$ and similar to $\pmatrix{B_{11}&0\\ 0&0}$. In fact, we can exhibit the similarity transforms explicitly:
\begin{aligned}
&\pmatrix{I&B_{11}^{-1}B_{12}\\ 0&I}
\pmatrix{B_{11}&B_{12}\\ 0&0}
\pmatrix{I&-B_{11}^{-1}B_{12}\\ 0&I}\\
&=\pmatrix{B_{11}&0\\ 0&0}\\
&=\pmatrix{I&0\\ -B_{21}B_{11}^{-1}&I}
\pmatrix{B_{11}&0\\ B_{21}&0}
\pmatrix{I&0\\ B_{21}B_{11}^{-1}&I}.
\end{aligned}
</s>