<s>[INST] Evaluation of $\int_0^{\pi/4} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx$ How to evaluate the following integral
$$\int_0^{\pi/4} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx$$
It looks like beta function but Wolfram Alpha cannot evaluate it. So, I computed the numerical value of integral above to 70 digits using Wolfram Alpha and I used the result to find its closed-form. The possible candidate closed-form from Wolfram Alpha is
$$\pi\sqrt{\frac{1+\sqrt{2}}{2}}-\pi$$
Is this true? If so, how to prove it?
 [/INST] To get rid of both square root, I use the substitution$ \tan x = \sin^2\theta$.  Then we rewrite the integral I as \begin{array}{l}
\displaystyle I=\int_{0}^{\frac{\pi}{2}} \sin \theta \sqrt{1-\sin ^{2} \theta} \cdot\frac{2 \sin \theta \cos \theta d \theta}{1+\sin ^{4} \theta} \\
=2 \displaystyle  \int_{0}^{\frac{\pi}{2}} \dfrac{\sin ^{2} \theta \cos ^{2} \theta}{1+\sin ^{4} \theta} d \theta.
\end{array}
Applying the identity $\cos^2x=1-\sin^2x$ yields
$$I= 2 \underbrace{\int_{0}^{\frac{\pi}{2}} \frac{1+\sin ^{2} \theta}{1+\sin ^{4} \theta} d \theta}_{J}-\pi $$
$$
J=2 \int_{0}^{\frac{\pi}{2}} \frac{\sec ^{2} \theta\left(\sec ^{2} \theta+\tan ^{2} \theta\right)}{\sec ^{4} \theta+\tan ^{4} \theta} d \theta$$
Letting $\tan \theta \mapsto t$, we have
$$J=2 \int_{0}^{\infty} \frac{1+2 t^{2}}{2 t^{4}+2 t^{2}+1} d t \\
=\int_{0}^{\infty} \frac{1+2 t^{2}}{t^{4}+t^{2}+\frac{1}{2}} d t\\ 
=\int_{0}^{\infty} \frac{2+\frac{1}{t^{2}}}{t^{2}+\frac{1}{2 t^{2}}+1} d t$$
Using the identity $\displaystyle 2+\frac{1}{t^{2}}=\frac{2+\sqrt{2}}{2}\left(1+\frac{1}{\sqrt{2} t^{2}}\right)+\frac{2-\sqrt{2}}{2}\left(1-\frac{1}{\sqrt{2} t^{2}}\right)$ yields
$$\displaystyle J=\frac{2+\sqrt{2}}{2} \int_{0}^{\infty} \frac{d\left(t-\frac{1}{\sqrt{2} t}\right)}{\left(t-\frac{1}{\sqrt{2}t}\right)^{2}+(1+\sqrt{2})}+\frac{2-\sqrt{2}}{2} \int_{0}^{\infty} \frac{d\left(t+\frac{1}{\sqrt{2} t}\right)}{\left(t+\frac{1}{\sqrt{2}t}\right) ^{2} -(\sqrt{2}-1)}$$
$$
=\frac{2 +\sqrt 2}{2 \sqrt{1+\sqrt{2}}} \left[\tan ^{-1}\left(\frac{t-\frac{1}{\sqrt{2} t}}{\sqrt{1+\sqrt{2}}}\right)\right]_{0}^{\infty}+\frac{2-\sqrt{2}}{4 \sqrt{\sqrt{2}-1}} \ln \left|\frac{t^{2}-t  \sqrt{\sqrt{2}-1}+1}{t^{2}+t \sqrt{\sqrt{2}-1}+1}\right|_{0}^{\infty} \\ $$
$ \displaystyle \qquad\qquad =\frac{2+\sqrt{2}}{2 \sqrt{1+\sqrt{2}}} \pi \\
\displaystyle \qquad\qquad =\sqrt{\frac{1}{2}+\frac{1}{\sqrt{2}}} \pi
$
Finally, we can conclude that $$I= 
\left(\sqrt{\frac{1}{2}+\frac{1}{\sqrt{2}}}-1\right) \pi
$$
</s>