<s>[INST] A question on cosine integral So I've read a book and found myself stumped in this integral:

$$\int_{0}^{\pi} \frac{\cos(n\theta)}{b^2-a^2\cos(2\theta)}\, d\theta=\begin{cases}
\,\,0 &,\quad\mbox{if}\,\, n\,\,\mbox{is odd}\\[20pt]
\,\,\dfrac{\pi}{\sqrt{b^4-a^4}}\left(\dfrac{\sqrt{b^2-\sqrt{b^4-a^4}}}{a}\right)^n&,\quad\mbox{if}\,\, n\,\,\mbox{is even}\\
\end{cases}$$
  where $b>a$.

Anyone knows how to evaluate it? Or knows a reference for helping me to prove formula above?
 [/INST] If $n=2k+1$, then
$$\int_{\pi/2}^\pi\frac{\cos((2k+1)\theta)}{b^2-a^2\cos(2\theta)}d\theta=\int_0^{\pi/2}\frac{\cos((2k+1)(\pi-\theta))}{b^2-a^2\cos(2(\pi-\theta))}d\theta=\int_0^{\pi/2}\frac{-\cos((2k+1)\theta)}{b^2-a^2\cos(2\theta)}d\theta$$
and the integral is $0$ because the convergence is trivial. So assume $n=2k$, then
$$\int_0^\pi\frac{\cos(2k\theta)}{b^2-a^2\cos(2\theta)}d\theta=\int_0^{2\pi}\frac{\cos(k\theta)}{2b^2-2a^2\cos\theta}d\theta=\frac1{a^2}\int_0^{2\pi}\frac{\cos(k\theta)}{2c-2\cos\theta}d\theta,$$
where $c=b^2/a^2>1$. Now
\begin{align*}\int_0^{2\pi}\frac{\cos(k\theta)}{2c-2\cos\theta}d\theta&=\text{Re}\int_0^{2\pi}\frac{e^{ik\theta}}{2c-(e^{i\theta}+e^{-i\theta})}d\theta=\text{Re}\frac1i\int_0^{2\pi}\frac{-e^{ik\theta}\cdot ie^{i\theta}}{e^{2i\theta}-2ce^{i\theta}+1}d\theta\\&=\text{Re}\frac1i\int_\varphi\frac{-z^k}{z^2-2cz+1}dz,\end{align*}
where $\varphi(\theta)=e^{i\theta}$ on $[0,2\pi]$. The denominator has 2 roots and only $\gamma=c-\sqrt{c^2-1}$ is inside $\text{Int}\,\varphi$, the residue is
$$\frac{-\gamma^k}{2\gamma-2c}=\frac{(c-\sqrt{c^2-1})^k}{2\sqrt{c^2-1}}.$$
Putting all things together yields the answer
$$\frac1{a^2}\text{Re}\frac{2\pi i}i\frac{(c-\sqrt{c^2-1})^k}{2\sqrt{c^2-1}}=\frac{\pi(c-\sqrt{c^2-1})^k}{a^2\sqrt{c^2-1}}=\frac{\pi\Big(b^2-\sqrt{b^4-a^4}\Big)^k}{a^{2k}\sqrt{b^4-a^4}}$$
</s>