<s>[INST] How to evaluate the integral $\int_0^{\pi/2}x^2(\sin x+\cos x)^3\sqrt{\sin x\cos x} \, dx$? How to evaluate the integral $$\int_0^{\pi/2}x^2(\sin x+\cos x)^3(\sin x\cos x)^{1/2} \, dx \text{ ?}$$
I tried to subsititution $x=\frac{\pi}{2}-t$, but it doesn't work. can someone
help me, any hint or trick are appreciated.
 [/INST] $$\color{blue}{\int_0^{\pi/2} {{x^2}{{(\sin x + \cos x)}^3}\sqrt {\sin x\cos x} dx} = \frac{\pi}{384\sqrt{2}}\left(35 \pi ^2-84 \ln ^22+132 \ln 2-150 \right)}$$

Note that $$\int_0^{\pi/2} {{x^2}{{(\sin x + \cos x)}^3}\sqrt {\sin x\cos x} dx}  = \int_0^{\pi/2} {{x^2}{{(\sin x + \cos x)}^3}\sin x\sqrt {\cot x} dx}$$
and the identity
$$(\sin x + \cos x)^3 \sin x = \frac{1}{4} (4 \sin (2 x)-\sin (4 x)-2 \cos (2 x)-\cos (4 x)+3)$$
Thus it suffices to find
$$I_1 = \int_0^{\pi/2} {{x^2}\sqrt {\cot x} dx}
\qquad I_2 = \int_0^{\pi/2} {{x^2}\sqrt {\cot x} \cos (2x)dx} \qquad 
I_3 = \int_0^{\pi/2} {{x^2}\sqrt {\cot x} \cos (4x)dx}$$
$$I_4 = \int_0^{\pi/2} {{x^2}\sqrt {\cot x} \sin (2x)dx}\qquad
I_5 = \int_0^{\pi/2} {{x^2}\sqrt {\cot x} \sin (4x)dx}$$


Whenever the integral converges,
  $$\tag{1}\int_0^\pi \frac{e^{i ax}}{\sin^b x} dx = \frac{{{2^b}\Gamma (1 - b)\pi }}{{\Gamma (1 + \frac{{a - b}}{2})\Gamma (1 - \frac{{a + b}}{2})}}{e^{ia\pi /2}}$$ this is a direct consequence of contour integration

Now, note that
$$\begin{aligned}\int_0^{\frac{\pi }{2}} {{x^2}\sqrt {\cot x} \cos rxdx}  &= \sqrt 2 \int_0^{\frac{\pi }{2}} {{x^2}\frac{{\cos x}}{{\sqrt {\sin 2x} }}\cos rxdx}  \\&= \frac{{\sqrt 2 }}{8}\int_0^\pi  {{x^2}\frac{{\cos \frac{x}{2}}}{{\sqrt {\sin x} }}\cos \frac{{rx}}{2}dx}  \\&= \frac{{\sqrt 2 }}{{16}}\int_0^\pi  {\frac{{{x^2}}}{{\sqrt {\sin x} }}\left[ {\cos \frac{{(1 + r)x}}{2} + \cos \frac{{(1 - r)x}}{2}} \right]dx}\end{aligned}$$
the RHS can be evaluated by taking $b=1/2$ then differentiates $(1)$ with respect to $a$ twice.
Substituting suitable values of $r$ gives
$$I_1 = \frac{{\sqrt 2 \pi }}{{96}}(5{\pi ^2} - 12\pi \ln 2 - 12{\ln ^2}2)$$
$$I_2 = \frac{\pi}{96\sqrt{2}} \left(5 \pi ^2-12 \pi +24-12 \ln ^22-12 \pi  \ln 2-24 \ln 2 \right)$$
$$I_3 = \frac{\pi}{192\sqrt{2}} \left(5 \pi ^2-18 \pi +54-12 \ln ^22-12 \pi  \ln 2-36 \ln 2 \right)$$
Similar trick works for $I_4, I_5$, with respective values:
$$I_4 = \frac{\pi}{96\sqrt{2}} \left(5 \pi ^2-12 \pi -24-12 \ln ^22+12 \pi  \ln 2+24 \ln 2 \right)$$
$$I_5 = \frac{\pi}{196\sqrt{2}} \left(5 \pi ^2-30 \pi -42-12 \ln ^22+12 \pi  \ln 2+60 \ln 2 \right)$$
</s>