<s>[INST] Prove $\int_0^{\pi/2}x^2\sqrt{\cot x} \ dx=\frac{\sqrt2}{8}\left( \frac{5\pi^{3}}{12}-\pi^2\ln2-\pi \ln^22 \right)$ I came across the follwing integral:
$$\int_0^{\pi/2}x^2\sqrt{\cot x} \ dx=\frac{\sqrt2}{8}\left( \frac{5\pi^{3}}{12}-\pi^2\ln2-\pi \ln^22 \right)$$
I try to do it like the following:
Consider
$$I(a,b)=\int_0^{\pi} \frac{\cos ax}{\sin^b x}\ dx=2\int_0^{\pi/2} \frac{\cos 2ax}{\sin^b 2x}\ dx$$
Then
$$I''(a,b)=-8\int_0^{\pi/2}x^2 \frac{\cos 2ax}{\sin^b 2x}\ dx$$
Let $a=1/2,b=1/2$
$$I''(1/2,1/2)=-8\int_0^{\pi/2}x^2 \frac{\cos x}{\sqrt{\sin 2x}}\ dx=-\frac{8}{\sqrt2}\int_0^{\pi/2}x^2\sqrt{\cot x} \ dx $$
Back to $I(a,b),\quad I(a,b) $can be expressed by beta fuction by Wolfram Mathematica.
$$I(a,b)=\int_0^{\pi} \frac{\cos ax}{\sin^b x}\ dx=\frac{\pi \cdot 2^b\cdot\cos (\pi a/2)\cdot \Gamma(1-b) }{\Gamma(a/2-b/2+1) \cdot \Gamma(-a/2-b/2+1)} dx$$
Then we can get $I''(a,b)$ approach from other way.
Finally,we can get $\int_0^{\pi/2}x^2\sqrt{\cot x} \ dx$,but it seems a little complex. 
For a similar integral: 
$$\int_0^{\pi/2}x\cdot\tan^p x \ dx=\frac{\pi}{4\sin (p\pi/2)}\left(\Psi\left(\frac{1}{2}\right)-\Psi\left(\frac{1-p}{2}\right) \right)$$
The above integral can be solved by method of parametric development.Let $p=-\frac{1}{2}$,We can get 
$$\int_0^{\pi/2}x\sqrt{\cot x} \ dx=\frac{\pi\left(\pi-2\ln 2\right)}{4\sqrt2}$$
But to this one, it seems to be difficult with method of parametric development.  Could you suggest any ideas how to prove this?
 [/INST] A slightly easier form for $I(a, \frac{1}{2})$ is
$$I(a, \frac{1}{2}) = 2^{a-\frac{3}{2}} \beta\left(\frac{1+2a}{4}\right) (1 + \sin \pi a + \cos \pi a), $$
where $\beta(p) = \Gamma(p)^2/\Gamma(2p)$ is the central beta function. (This can be easily obtained by applying the Euler reflection formula and the Legendre duplication formula to OP's representation.) Now from the relation
$$ \frac{d}{dp} \log \beta(p) \bigg|_{p=\frac{1}{2}} = 2\psi(1/2) - 2\psi(1) = 2 \sum_{n=0}^{\infty} \left( \frac{1}{n+1} - \frac{1}{n+\frac{1}{2}} \right) = -4 \log 2 $$
and
$$ \frac{d^2}{dp^2} \log \beta(p) \bigg|_{p=\frac{1}{2}} = 2\psi^{(1)}(1/2) - 4\psi^{(1)}(1) = \frac{\pi^2}{3}, $$
we can compute $\frac{\partial^2 I}{\partial a^2}(\frac{1}{2}, \frac{1}{2})$ as follows:
\begin{align*}
&\frac{\partial^2 I}{\partial a^2} (1/2, 1/2) \\
&= \frac{1}{4} \beta''(1/2) - \left(\frac{\pi}{2}-\log 2\right) \beta'(1/2) - \left(\frac{\pi^2}{2} + \pi\log 2-\log^2 2\right) \beta(1/2) \\
&= \frac{1}{4} \pi \left( 16\log^2 2 + \frac{\pi^2}{3} \right) \\
&\qquad - \left(\frac{\pi}{2}-\log 2\right) (-4\pi \log 2)
 - \left(\frac{\pi^2}{2} + \pi\log 2-\log^2 2\right) \pi \\
&= -\left(\frac{5\pi^3}{12} - \pi^2\log2 - \pi \log^2 2 \right). 
\end{align*}

I am also trying a more direct approach. Using contour integral, we can check that
$$ \int_{0}^{\frac{\pi}{2}} x^2 \sqrt{\cot x} \, \mathrm{d}x
= \frac{1}{3\sqrt{2}} \left( \frac{\pi^3}{4} - \frac{3\pi}{2} \int_{0}^{1} \frac{\operatorname{artanh}^2 t}{\sqrt{t}} \, \mathrm{d}t \right). $$
So I am tackling the last integral, but have no good news at this point.
</s>