<s>[INST] Prove that $\int_0^\pi\frac{\cos x \cos 4x}{(2-\cos x)^2}dx=\frac{\pi}{9} (2160 - 1247\sqrt{3})$ Prove that
$$\int_0^\pi\frac{\cos x \cos 4x}{(2-\cos x)^2}dx=\frac{\pi}{9} (2160 - 1247\sqrt{3})$$
I tried to use Weierstrass substitution but the term $\cos 4x$ made horrible algebraic-forms since $\cos 4x = \sin^4 x + \cos^4 x - 6\sin^2 x \cos^2 x$. My friend suggests me use a contour integration method but I am not familiar with that method. Any idea? Any help would be appreciated. Thanks in advance.
 [/INST] I would like to try to tackle the problem with elementary integration techniques though it is a bit tedious. First of all, using double angle formula, we get
$$\cos x\cos 4x= 8 \cos ^{5} x-8 \cos ^{3} x+\cos x$$ and hence
$$
I=\int_{0}^{\pi} \frac{8 \cos ^{5} x-8 \cos ^{3} x+\cos x}{(2-\cos x)^{2}} d x
$$
By division, we reduce the integrand to a proper fraction,
$$
\begin{aligned}
I \displaystyle &=\int_{0}^{\pi}\left[8 \cos ^{3} x+32 \cos ^{2} x+88 \cos x+224\right] d x + \displaystyle \int_{0}^{\pi} \frac{545 \cos x-896}{(\cos x-2)^2} d x \\&= 240 \pi+545\underbrace{\int_{0}^{\pi} \frac{d x}{\cos x-2}}_{J}+194 \underbrace{\int_{0}^{\pi} \frac{d x}{(\cos x-2)^{2}}}_{K} 
\end{aligned}
$$
For any $a>1,$ let’s define a definite integral$$
\begin{aligned}
\displaystyle I(a):&=\int_{0}^{\pi} \frac{d x}{a-\cos x} \stackrel{x\mapsto \pi-x}{=} \int_{0}^{\pi} \frac{d x}{a+\cos x}\\
\\2 I(a)&=\int_{0}^{\pi}\left(\frac{1}{a-\cos x}+\frac{1}{a+\cos x}\right) d x\\
&=4 a \int_{0}^{\frac{\pi}{2}} \frac{1}{a^{2}-\cos ^{2} x} d x\\
&=4 a \int_{0}^{\frac{\pi}{2}} \frac{\sec ^{2} x}{a^{2} \sec ^{2} x-1} d x\\
&=4 a  \int_{0}^{\frac{\pi}{2}} \frac{d(\tan x)}{a^{2} \tan ^{2} x+\left(a^{2}-1\right)}\\
&=\frac{4}{\sqrt{a^{2}-1}}\left[\tan ^{-1}\left(\frac{a \tan x}{\sqrt{a^{2}-1}}\right)\right]_{0}^{\frac{\pi}{2}}\\
&=\frac{2\pi}{\sqrt{a^{2}-1}}\\
\therefore \quad \int_{0}^{\pi} \frac{d x}{a-\cos x}&=\frac{\pi}{\sqrt{a^{2}-1}} 
\end{aligned}
$$
In particular, $$J=-I(2)=-\frac{\pi}{\sqrt{3}}$$
For $K$, we just differentiate $I(a)$ w.r.t. $a$.
$$
\begin{aligned}
I^{\prime}(a)=-\frac{\pi}{\left(a^{2}-1\right)^{\frac{3}{2}}} 
&\Rightarrow \int_0^{\pi}\frac{1}{(a-\cos x)^{2}} d x=\frac{\pi a}{\left(a^{2}-1\right)^{\frac{3}{2}}} \\
\therefore K=\int_{0}^{\pi} \frac{1}{(2-\cos x)^{2}} d x&=\frac{2 \pi}{3 \sqrt{3}} \\
\end{aligned}
$$
Now we can conclude that
$$I =240 \pi+545\left(-\frac{\pi}{\sqrt{3}}\right)+194\left(\frac{2 \pi}{3 \sqrt{3}}\right) =\frac{\pi}{9}(2160 - 1247 \sqrt{3})$$
</s>