<s>[INST] Prove that $\int_0^\pi\frac{\cos x \cos 4x}{(2-\cos x)^2}dx=\frac{\pi}{9} (2160 - 1247\sqrt{3})$ Prove that
$$\int_0^\pi\frac{\cos x \cos 4x}{(2-\cos x)^2}dx=\frac{\pi}{9} (2160 - 1247\sqrt{3})$$
I tried to use Weierstrass substitution but the term $\cos 4x$ made horrible algebraic-forms since $\cos 4x = \sin^4 x + \cos^4 x - 6\sin^2 x \cos^2 x$. My friend suggests me use a contour integration method but I am not familiar with that method. Any idea? Any help would be appreciated. Thanks in advance.
 [/INST] Proposition : 

\begin{equation}\int_0^\pi\frac{\cos mx}{p-q\cos x}\, dx=\frac{\pi}{\sqrt{p^2-q^2}}\left(\frac{p-\sqrt{p^2-q^2}}{q}\right)^m\qquad\hbox{for}\qquad |q|<p
\end{equation}


Proof :
We have
\begin{equation}
\int_0^\pi\frac{\cos mx}{a^2-2ab\cos x+b^2}\, dx=\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m\qquad\hbox{for}\qquad |b|<a\tag1
\end{equation}
The complete proof is given by Prof. Omran Kouba and can be seen here.
Now, let $p=a^2+b^2$ and $q=2ab$, then $p+q=\sqrt{p+q}$ and $p-q=\sqrt{p-q}$. Therefore
\begin{align}
2a&=\sqrt{p+q}+\sqrt{p-q}\\[10pt]
2b&=\sqrt{p+q}-\sqrt{p-q}\\[10pt]
a^2-b^2&=\sqrt{p^2-q^2}\tag2\\[10pt]
\frac{b}{a}&=\frac{p-\sqrt{p^2-q^2}}{q}\tag3
\end{align}
then plugging in $(2)$ and $(3)$ to $(1)$ we prove our proposition. $\quad\square$
Set $m=4$ and $p=2$ then differentiate the proposition w.r.t. $q$ and take the limit for $q\to1$, we obtain
\begin{align}
\lim_{q\to1}\int_0^\pi\partial_q\left(\frac{\cos 4x}{2-q\cos x}\right)\, dx&=\lim_{q\to1}\partial_q\left(\frac{\pi}{\sqrt{4-q^2}}\left(\frac{2-\sqrt{4-q^2}}{q}\right)^4\right)\\[10pt]
\int_0^\pi\frac{\cos x \cos 4x}{(2-\cos x)^2}dx&=\frac{\pi}{9} \left(\,2160 - 1247\sqrt{3}\,\right)
\end{align}
The last step is confirmed by Wolfram Alpha.
I think differentiating is easier than using contour integration or partial fraction decomposition. (>‿◠)✌
</s>