<s>[INST] Given that $a>b>0$, can we find $\int_{0}^{2 \pi} \frac{\cos n x}{a+b \cos x} d x$ and $\int_{0}^{2 \pi} \frac{\sin n x}{a+b \cos x} d x?$ Let’s first consider
$$
\int_{0}^{2 \pi} \frac{\cos n x}{a+b \cos x} d x+i \int_{0}^{2 \pi} \frac{\sin n x}{a+b \cos x} d x = \int_{0}^{2 \pi} \frac{e^{n x i}}{a+b \cos x} dx=I \tag*{(*)} 
$$
where $a>b>0$.
Next we are going to evaluate $I$ using contour integration by letting $z=e^{xi}$.  Then
$$
 I= \oint_{|z|=1} \frac{z^{n}}{a+\frac{b\left(z+z^{-1}\right)}{2}} \cdot \frac{d z}{i z} =\frac{2}{i} \oint_{|z|=1} \frac{z^{n}}{b(z-\alpha)(z-\beta)} d z,
$$
where $\alpha=\frac{-a+\sqrt{a^{2}-b^{2}}}{b}$, $\beta=\frac{-a-\sqrt{a^{2}-b^{2}}}{b}$  with  $|\beta| =\frac{a+\sqrt{a^{2}-b^{2}}}{b} > \frac{a}{b}> 1 \Rightarrow |\alpha |<1$.
Therefore $\alpha $ is the unique simple pole in the unit circle $|z|=1.$ It’s residue is
$$\operatorname{Res}\left(\frac{z^{n}}{b(z-\alpha)(z-\beta)}, z=\alpha \right)=\lim _{z \rightarrow \alpha} \frac{z^{n}}{b(z-\beta)}=\frac{\alpha^{n}}{2 \sqrt{a^{2}-b^{2}}}$$
Hence
$$
\begin{aligned}
I &=\frac{2}{i} \cdot 2 \pi i \cdot \frac{\alpha^{n}}{2 \sqrt{a^{2}-b^{2}}}=\frac{2 \pi \alpha^{n}}{\sqrt{a^{2}-b^{2}}}\in R.
\end{aligned}
$$
Comparing the real and imaginary parts of both sides in (*) yields the result.
$$
\boxed{\int_{0}^{2 \pi} \frac{\cos n x}{a+b \cos x} d x=\frac{2 \pi}{\sqrt{a^{2}-b^{2}}}\left(\frac{-a+\sqrt{a^{2}-b^{2}}}{b}\right)^{n}
\textrm{  and }\int_{0}^{2 \pi} \frac{\sin n x}{a+b \cos x} d x=0}$$
Furthermore, if $a>-b>0$, then
$$\int_{0}^{2 \pi} \frac{\cos n x}{a+b \cos x} d x=\frac{2 \pi}{\sqrt{a^{2}-b^{2}}}\left(\frac{-a-\sqrt{a^{2}-b^{2}}}{b}\right)^{n}
\textrm{  and }\int_{0}^{2 \pi} \frac{\sin n x}{a+b \cos x} d x=0 $$
I really want to know whether there are other methods. Your comments and alternative solutions are highly appreciated!
 [/INST] Utilize the Fourier series
$$\eqalign{
\frac{1-r^2}{1+r^2+2r\cos x}=1+2\sum_{k>0}(-r)^k\cos k x
}
$$
and let $r=\frac {a-\sqrt{a^2-b^2}}b$ to integrate
\begin{align}
&\int_{0}^{2 \pi} \frac{e^{i n x}}{a+b \cos x} dx\\
=&\  \frac1{\sqrt{a^2-b^2}}\int_{0}^{2 \pi} \frac{(1-r^2) e^{i n x}}{1+r^2+2r\cos x} dx\\
=&\ \frac1{\sqrt{a^2-b^2}}\int_0^{2\pi}
e^{i n x}\bigg( 1+2\sum_{k>0}(-r)^k\cos k x\bigg)dx\\
= &\ \frac1{\sqrt{a^2-b^2}}\int_0^{2\pi}(-r)^n dx
=\frac{2 \pi \ (-r)^n}{\sqrt{a^{2}-b^{2}}}
\end{align}
whose real and imaginary parts are, respectively
$$\int_{0}^{2 \pi} \frac{\cos n x}{a+b \cos x} d x=\frac{2 \pi(-r)^n}{\sqrt{a^{2}-b^{2}}};\>\>\>\>\>
\int_{0}^{2 \pi} \frac{\sin n x}{a+b \cos x} d x=0 $$
</s>