<s>[INST] Conjugating simultaneously two matrices to an integer matrix I have the matrices $$A=\begin{pmatrix} 1&0&0&0\\0&1&0&0\\0&0&-1&0\\0&0&0&-1 \end{pmatrix},\quad B=\begin{pmatrix} 1&0&0&0\\0&1&0&0\\0&0&\cos\frac{2\pi}{3}&-\sin\frac{2\pi}{3}\\0&0&\sin\frac{2\pi}{3}&\cos\frac{2\pi}{3} \end{pmatrix}.$$
Problem: I'm trying to decide if there exists some $P\in\mathsf{GL}(4,\mathbb{R})$ such that $P^{-1} A P=\begin{pmatrix} 1&0&0&0\\0&-1&0&0\\0&0&1&1\\0&0&0&-1\end{pmatrix}=:E$ and s.t $P^{-1}BP$ be an integer matrix. Note that $B^3=I_3$.
Thoughts: I know that there exists some $P\in\mathsf{GL}(4,\mathbb{R})$ s.t $P^{-1}AP=E$ and such $P$ must be of the form $P=\begin{pmatrix} a&0&2c&c\\b&0&2d&d\\0&e&0&f\\0&g&0&h\end{pmatrix}$.
But there are many equations involved if I try to set $P^{-1}BP=F$ and try to solve. Also, I know that there are three conjugacy classes of order 3 in $\mathsf{GL}(4,\mathbb{Z})$: $$\begin{pmatrix} 0&-1&0&0\\1&-1&0&0\\0&0&0&-1\\0&0&1&-1\\ \end{pmatrix},\begin{pmatrix} 1&0&0&0\\0&1&0&0\\0&0&0&-1\\0&0&1&-1\end{pmatrix},\begin{pmatrix} 1&0&1&0\\0&1&0&0\\0&0&0&-1\\0&0&1&-1\end{pmatrix}$$ and $B$ can be $\mathsf{GL}(4,\mathbb{R})$-conjugated (and only) to the last two.
I don't know if maybe another approach could be useful. Could you give me some help or ideas? Thanks
 [/INST] No such $P$ exists. Suppose the contrary. Let
$$
X=P^{-1}AP=\pmatrix{1&0&0&0\\ 0&-1&0&0\\ 0&0&1&1\\ 0&0&0&-1},
\ Y=P^{-1}BP=\pmatrix{a&b&c&d\\ e&f&g&h\\ i&j&k&l\\ m&n&o&p}.
$$
Since $AB=BA$, we have $XY=YX$, i.e.
$$
\pmatrix{a&b&c&d\\ -e&-f&-g&-h\\ i+m&j+n&k+o&l+p\\ -m&-n&-o&-p}
=\pmatrix{a&-b&c&c-d\\ e&-f&g&g-h\\ i&-j&k&k-l\\ m&-n&o&o-p}.
$$
By comparing coefficients on both sides, we obtain
$$
Y=\pmatrix{a&0&2d&d\\ 0&f&0&h\\ i&j&2l+p&l\\ 0&-2j&0&p}.
$$
By swapping the two middle rows and the two middle columns of $Y$, we see that $Y$ is similar to
$$
Z=\left(\begin{array}{cc|cc}a&2d&0&d\\ i&2l+p&j&l\\ \hline 0&0&f&h\\ 0&0&-2j&p\end{array}\right)=\pmatrix{S&\ast\\ 0&T}.
$$
Hence $Z$ is similar to $B$, the submatrices $S$ and $T$ are integer cube roots of $I_2$ and one of them is similar to the rotation matrix $R$ for an angle $2\pi/3$. Yet this is impossible, because $R$ has trace $-1$ and determinant $1$. Every integer $2\times2$ matrix with an odd trace and an odd determinant must have an odd anti-diagonal, but $S$ has an even anti-diagonal entry $2d$ and $T$ has an even anti-diagonal entry $-2j$.
</s>