<s>[INST] Let A,B be two $4\times4$ real commuting matrices and $\det(A^2-AB+B^2) = 0$ show that $\det(A+B) + 3\det(A-B) = 6\det(A) + 6\det(B)$ Let $A$, $B$ be two $4\times 4$ matrices with real enteries, such that $AB = BA$ and $\det(A^2-AB+B^2) = 0$. Show that $\det(A+B) + 3\det(A-B) = 6\det(A) + 6\det(B)$.
How can I get started on this question? 
 [/INST] We can exploit the fact that each expression is homogeneous on $A$ and $B$.
Assume that $\det(B) \neq 0$. Multiply both sides of the first equation by $\det(B)^{-2}$, and both sides of the second equation by $\det(B)^{-1}$.  Using the fact that $A$ and $B$ commute, we have
$$
\det(A^2 - AB + B^2)\det(B)^{-2} = 0 \implies\\
\det(B^{-2}(A^2 - AB + B^2)) = 0 \implies\\
\det((AB^{-1})^2 - (AB^{-1}) + I) = 0.
$$
Similarly, multiply both sides of the second equation by $\det(B)^{-1}$ to get
$$
\det((AB^{-1}) + I) + 3 \det((AB^{-1}) - I) = 6 \det(AB^{-1}) + 6.
$$
So, let $X = AB^{-1}$. It suffices to show that 
$$
\det(X^2 - X + I) = 0 \implies \det(X + I) + 3 \det(X - I) = 6 \det(X) + 6.
$$
From there, the trick is hidden below (put your mouse over it to see).  This is enough to get started though, and I recommend that you try to figure this part out for yourself.
Hint:

 $X$ is a matrix with real entries, so its complex eigenvalues must come in conjugate pairs.  

Solution:

Thus, we can guarantee that the roots of $x^2 - x + 1 = 0$, 
 namely $\lambda_\pm = \frac 12 \pm \frac{\sqrt{3}}2 i$, must be eigenvalues of 
 $X$.  Let $a,b$ denote the remaining two eigenvalues.  We can now rewrite each 
 expression as follows:

 $$ \det(X+I) = (\lambda_+ +1)(\lambda_- + 1)(a+1)(b+1) = 3(a+1)(b+1)\\ \det(X-I) = (\lambda_+ - 1)(\lambda_- - 1)(a-1)(b-1) = (a-1)(b-1)\\ \det(X) = \lambda_+ \lambda_- ab = ab$$

 We can now show by routine simplification that the desired equality holds in the case that $\det(B) \neq 0$.

For the case that $\det(B) = 0$, it suffices to makes an argument by continuity.  Since 
$$
\det(A+(B + \epsilon I)) + 3\det(A-(B + \epsilon I)) = 6\det(A) + 6\det(B + \epsilon I)
$$
will hold for all sufficiently small $\epsilon > 0$ (which will force $B + \epsilon I$ to be invertible), it must also hold that the two sides are equal for $\epsilon = 0$.
</s>