<s>[INST] How to prove(or disprove) $\begin{vmatrix} A&B\\ B&A \end{vmatrix}=|A^2-B^2|$ Let $A$ and $B$ be square matrices of the same size.
(1) If $f$ is not invertible and $AB=BA$, show that
$$\begin{vmatrix}
A&B\\
B&A
\end{vmatrix}=|A^2-B^2|.$$
(2) If $A$ is invertible and $AB\neq BA$, then do we have
$$\begin{vmatrix}
A&B\\
B&A
\end{vmatrix}=|A^2-B^2|?$$
My try: I have only proved this: if $A$ is invertible and $AB=BA$, then
$$\begin{vmatrix}
A&B\\
B&A
\end{vmatrix}=|A^2-B^2|.$$
For $(1)$ and $(2)$, I can't prove. Thank you. 
 [/INST] Let
$$
J=\left(\begin{array}{cc}I&0\\0&-I\end{array}\right)
$$
and
$$
M(A,B)=\left(\begin{array}{cc}A&B\\B&A\end{array}\right).
$$
Then $\det J^2=\det I=1$, so $\det(JM(A,B)J)=\det M(A,B)$. But (check this)
$$JM(A,B)J=M(A,-B),$$
so $\det M(A,B)=\det M(A,-B)$.
On the other hand using $AB=BA$ we get
$$
M(A,B)M(A,-B)=\left(\begin{array}{cc}A^2-B^2&-AB+BA\\BA-AB&A^2-B^2\end{array}\right)=\left(\begin{array}{cc}A^2-B^2&0\\0&A^2-B^2\end{array}\right).
$$
Putting these together we get
$$
\det M(A,B)^2=\left|\begin{array}{cc}A^2-B^2&0\\0&A^2-B^2\end{array}\right|.
$$
Can you take it from here?
</s>