<s>[INST] Prove that $\displaystyle\int_{0}^{\frac{1}{\sqrt{3}}}\frac{\arctan\left(\frac{1}{\sqrt{1-2x^2}}\right)}{1+x^2} \mathrm{dx}=\frac{13\pi^2}{288}$ I need to prove that,
$\displaystyle \tag*{} \int_{0}^{\frac{1}{\sqrt{3}}}\frac{\arctan\left(\frac{1}{\sqrt{1-2x^2}}\right)}{1+x^2} \mathrm{dx}=\frac{13\pi^2}{288}$
Here is what I tried:
I tried to solve the integral by Integrating by parts.
We have:
$\displaystyle \tag{1} \int f(x) g'(x) \mathrm{ dx} = f(x)g(x) - \int f'(x) g(x) \mathrm{ dx}$
I noticed ,
$\displaystyle \tag*{} \dfrac{1}{1+x^2} = \arctan '(x)$
$\displaystyle \tag*{} g'(x) = \dfrac{1}{1+x^2} \Leftrightarrow g(x) = \arctan(x)$
Now, I defined $f(x)$
$\displaystyle \tag*{} f(x) = \arctan \left( \dfrac{1}{\sqrt{1-2x^2}}\right )$
and
$\displaystyle \tag*{} f'(x) = \arctan'\left(\dfrac{1}{\sqrt{1-2x^2}} \right ) = \text{arccot}'(\sqrt{1-2x^2}) = \dfrac{x}{\sqrt{1-2x^2}(1-x^2)}$
Now, using $(1)$ and substituting the values of $f(x)$ and $g(x)$, My indefinite integral becomes:
$\displaystyle \tag*{} \arctan(x) \arctan \left (\dfrac{1}{\sqrt{1-2x^2}} \right ) - \int  \dfrac{x \arctan(x)}{\sqrt{1-2x^2}(1-x^2)} \mathrm{ dx}$
Now, we want to evaluate:
$\displaystyle \tag*{} \int \dfrac{x \arctan(x)}{\sqrt{1-2x^2}(1-x^2)} \mathrm{ dx}$
Now this is integral I am having trouble solving. Any hints or different methods would be greatly appreciated. Thank you.
 [/INST] A large portion of this answer is taken directly from this answer!

Enforce the substitution
$$y=\sqrt{1-2x^2}\qquad\qquad x^2=\frac {1-y^2}{2}\qquad\qquad\mathrm dx=-\frac {y}{\sqrt{2(1-y^2)}}\,\mathrm dy$$
The integral now becomes
$$\begin{align*}\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx & =\sqrt{2}\int\limits_{1/\sqrt{3}}^1\frac {y\left(\frac {\pi}2-\arctan y\right)}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\\ & =\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y\,\mathrm dy}{\sqrt{1-y^2}(3-y^2)}-\sqrt{2}\int\limits_{1/\sqrt3}^1\frac {y\arctan y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\end{align*}$$
Splitting up the integrand, the first integral can be evaluated with the substitution $y\mapsto\sqrt{1-y^2}$, giving
$$\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy=\frac {\pi^2}{12}$$
And enforcing the substitution $t=\sqrt{y}$ on the second integral, we have that
$$\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac 1{\sqrt{2}}\int\limits_{1/3}^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt$$
The last integral is difficult, but using the same methodology and formulas as the answer I have linked above, we first rewrite the integral such that the lower limit is zero.
$$\int\limits_{1/3}^1\frac {\arctan\sqrt t}{(3-t)\sqrt{1-t}}\,\mathrm dt=\underbrace{\int\limits_0^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{J}-\underbrace{\int\limits_0^{1/3}\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{K}$$
Next, we use the following formula to evaluate the right-hand side

$$\int\limits_0^x\frac {\arctan\sqrt t}{(a-t)\sqrt{b-t}}\,\mathrm dt=\frac 1{\sqrt{a-b}}S\left(\arctan\sqrt{\frac {b-x}{a-b}},\arctan\sqrt{\frac {b+1}{a-b}},\arctan\frac 1{\sqrt{a}}\right)$$

There are two important observations that will help us evaluate the two integrals, namely


*

*$S(0,\beta,\gamma)=\pi(\beta-\gamma)$

*When $\sin^2\alpha+\sin^2\gamma=\sin^2\beta$, then $S(\alpha,\beta,\gamma)=-\alpha^2+\beta^2-\gamma^2$

Substituting $a=3$ and $b=1$, then
$$\begin{align*}J & =\frac 1{\sqrt2}S\left(0,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{12\sqrt2}\\K & =\frac 1{\sqrt2}S\left(\frac {\pi}6,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{144\sqrt2}\end{align*}$$
Where I have used the first observation to evaluate $J$ and the second observation to evaluate $K$. Taking the difference $J-K$, then
$$\int\limits_{1/3}^1\frac {\arctan x}{(3-x)\sqrt{1-x}}\,\mathrm dx=\frac {11\pi^2}{144\sqrt2}$$
Putting everything together, we get that
$$\int\limits_0^{1/\sqrt{3}}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac {11\pi^2}{288}\color{blue}{=\frac {13\pi^2}{288}}$$
</s>