<s>[INST] By using properties of determinants show that determinant is equal to $(1+a^2+b^2)^3$ $$\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
2ab&1-a^2+b^2&2a\\
2b&-2a&1-a^2-b^2\end{vmatrix}=(1+a^2+b^2)^3$$
I have been trying to solve the above determinant. But unfortunately my answer is always coming as:
$$1+3a^2+3a^4+a^6+3a^2b^4+3b^2+4a^2b^2+a^4b^2+b^6+3b^4$$
 Please help me to solve this problem.
 [/INST] Assuming $a,b\in\mathbb{R}$ (and if not, proving it for real $a,b$ should allow you it to extend it to complex $a,b$), each row vector has the norm $(1+a^2+b^2)$ and the row vectors are all orthogonal so the matrix divided by $(1+a^2+b^2)$ is an orthogonal matrix and the absolute value of its determinant is 1 which immediately implies your result.
</s>