<s>[INST] About $\int_0^{\pi/2}\arctan(1-\sin^2 (x) \cos^2 (x))dx = \pi \left( \frac{\pi}{4}-\arctan \sqrt{\frac{\sqrt{2}-1}{2}}\right)$ In this question sos440 has mentioned about an integral that he computed:
$$\int_0^{\pi/2}\arctan(1-\sin^2 (x) \cos^2 (x))dx = \pi \left( \frac{\pi}{4}- \arctan \sqrt{\frac{\sqrt{2}-1}{2}}\right)$$
I would really like to know how this can be proved. I tried using differentiation under the integral sign on parmeter $a$:
$$\int_0^{\pi/2}\arctan(1-a\sin^2 (x) \cos^2 (x))dx $$
but it didn't really work. I would be really grateful if sos440 or somebody else can tell me the secret behind discovering this formula.
Thanks!
 [/INST] Updated (2019-08-19). Thanks to @darij grinberg I salvaged the content of my broken blog page. But I strongly recommend the reader to see @Pranav Arora's beautiful and neat computation first. This answer is more of 'brutal-force' solution.

Let $I$ denote this integral hereafter. To achieve this, we first observe by double-angle formula that
$$ \begin{aligned}
I
&= \int_{0}^{\frac{\pi}{2}} \arctan\left(1-\tfrac{1}{4}\sin^2 x\right) \, \mathrm{d}x \\
&= \frac{1}{2} \int_{0}^{\pi} \arctan\left(\tfrac{1}{8}(7+\cos x)\right) \, \mathrm{d}x \\
&=\frac{1}{4} \int_{-\pi}^{\pi} \arctan\left(\tfrac{1}{8}(7+\cos x)\right) \, \mathrm{d}x.
\end{aligned} $$
In the first and second step, we used the half-angle substitution $2x\mapsto x$. Then by symmetry, this is written as
Then by adopting the identity
$$ \int_{1}^{\infty} \frac{\alpha}{\alpha^2+t^2} \, \mathrm{d}t = \arctan(\alpha) \tag{$\alpha > 0$}, $$
the last identity for $I$ can be recast to
$$ I = 2 \int_{1}^{\infty} \int_{-\pi}^{\pi} \frac{7+\cos x}{(7+\cos x)^2 + 64t^2} \, \mathrm{d}x\mathrm{d}t. $$
Then we make change of variable $z = e^{ix}$ to obtain
$$ I = \frac{4}{i} \int_{1}^{\infty} \int_{\mathcal{C}} \frac{z^2 + 14z + 1}{(z^2 + 14z + 1)^2 + (16tz)^2} \, \mathrm{d}z\mathrm{d}t, $$
where $\mathcal{C}$ denote the counter-clockwised unit circle centered at the origin. Now let $f(z)$ denote the integrand of the above equality. To apply Cauchy integration formula, we have to find the pole inside the circle $\mathcal{C}$. To this end, let $\alpha_{\pm}, \beta_{\pm}$ denote
$$ \begin{gathered}
\alpha_{\pm} = -(7+8it) \pm\sqrt{(7+8it)^2-1}, \\
\beta_{\pm} = -(7-8it)\pm\sqrt{(7-8it)^2 - 1}, 
\end{gathered}$$
respectively. Note that $\alpha_{\pm}$ and $\beta_{\pm}$ are zeros of quadratic polynomials
$$z^2 + (14+16it)z + 1 \qquad \text{and} \qquad z^2 + (14-16it)z + 1$$
respectively, so that $f(z)$ factors into the form
$$ f(z) = \frac{z^2 + 14z + 1}{(z-\alpha_+)(z - \alpha_-)(z - \beta_+)(z - \beta_-)}. $$
Now for $t > 1$, only $\alpha_+$ and $\beta_+$ are contained in the inside of $\mathcal{C}$. Thus by Cauchy integration formula,
$$ \begin{aligned}
I
&= 8\pi \int_{1}^{\infty} \left( \underset{z=\alpha_+}{\mathrm{Res}} \, f(z) + \underset{z=\beta_+}{\mathrm{Res}} \, f(z) \right) \, \mathrm{d}t \\
&= 8\pi \int_{1}^{\infty} \left( \frac{1}{2(\alpha_+ - \alpha_-)} + \frac{1}{2(\beta_+ - \beta_-)} \right) \, \mathrm{d}t \\
&= 2\pi \int_{1}^{\infty} \left( \frac{1}{\sqrt{(7+8it)^2 - 1}} + \frac{1}{\sqrt{(7-8it)^2 - 1}} \right) \, \mathrm{d}t
\end{aligned} $$
To evaluate the integral in the right hand side, we note that
$$ \frac{d}{dz} \log(z+\sqrt{z^2-1}) = \frac{1}{\sqrt{z^2 - 1}}. $$
on the right-half plane $\{ z \in \mathbb{C} : \operatorname{Re}(z) > 0\}$. Then we may proceed as
$$\begin{aligned}
I
&= \frac{\pi}{4i} \left[ \log\left(7+8it + \sqrt{(7+8it)^2 - 1}\right) -  \log\left(7-8it + \sqrt{(7-8it)^2 - 1}\right) \right]_{1}^{\infty} \\
&= \frac{\pi}{2} \left[ \arg\left(7+8it + \sqrt{(7+8it)^2 - 1}\right) \right]_{1}^{\infty} \\
&= \frac{\pi}{2} \left( \frac{\pi}{2} - \arg \left( 7+8i + \sqrt{(7+8i)^2 - 1}\right) \right).
\end{aligned} $$
This horrifying argument term can be simplified further by noting that
$$ \frac{1}{2}\arg \left( 7+8i + \sqrt{(7+8i)^2 - 1}\right)
= \frac{1}{2}\arctan\left( \frac{\sqrt{8}}{7}\sqrt{5\sqrt{2}+1} \right)
= \arctan\sqrt{\frac{\sqrt{2}-1}{2}}. $$
</s>