<s>[INST] Contour integration problem with sin and cos so I'm revising contour integration for an upcoming complex analysis exam. I have been asked to integrate $$\int_0^{2\pi}\frac{\sin^2x}{a+b  \cos x}dx$$
I thought the sensible thing to do here would be to substitute in $z=e^{ix}$ and take a contour integral around the unit circle, call this path $\ast$ so that my integral becomes $$\frac{1}{2i}Re(\int_\ast\frac{1-z^2}{az+bz^2}dz)$$
Then, letting $f(z)=\frac{1-z^2}{az+bz^2}$, I thought the function had simple poles at $z=0$ with residue $\frac{1}{a}$ and another simple pole at $z=\frac{-a}{b}$ with residue $\frac{a}{b^2}-\frac{1}{a}$ and thus I get the that $$Re(\int_\ast\frac{1-z^2}{az+bz^2}dz)=2i(2\pi i)\frac{a}{b^2}=-4\pi(\frac{a}{b^2})$$ which is not the answer given, that is: $=\frac{2\pi}{b^2}[a-\sqrt{a^2-b^2}]$,but I can't work out why. 
Any help appreciated, thank you in advance. 
 [/INST] We suppose $b\ne 0$, and get rid of it by force, use rather instead of $a,b$ only the variable $c=a/b$.
(After some edits in the OP we have indeed $0<b<a$, thus $1<c$.)
We will assume $c$ real, $|c|>1$, so that there is no zero in the denominator. (And the integral does not diverge.)
Sooner or later we will have to fight against the two roots of $z^2+2c+1$, we denote them by $U,V$, their product is $UV=1$, and thus we may and do assume $|U|<1$, $|V|>1$. 

Let $C$ be the unit circle centered in the origin of the complex plane, then formally using $z=e^{ix}$, $\frac 1{iz}\; dz=dx$ we have:
$$
\begin{aligned}
J
&=
\int_0^{2\pi}\frac{\sin^2x}{a+b\cos x}\;dx
\\
&=
\frac 1b\int_0^{2\pi}\frac{\sin^2x}{\cos x+c}\;dx
\\
&=
\frac 1b
\int_C
\frac{
\left(\frac 1{2i}\left(z-\frac 1z\right)\right)^2}
{\frac 12\left(z+\frac 1z\right)+c}\;\frac 1{iz}\;dz
\\
&=
\frac i{2b}
\int_C
\frac{
(z^2-1)^2}
{z^2(z^2+2cz+1)}\;dz\ .
\\[3mm]
&\qquad\text{ The partial fraction decomposition is:}
\\
\frac{
(z^2-1)^2}
{z^2(z^2+2cz+1)}
&=
\frac{
(z^2-1)^2}
{z^2(z-U)(z-V)}
\\
&=
1+\frac 1{z^2}+\frac{U+V}z
-\frac  {U-V}{z-V}
-\frac  {V-U}{z-U}
\ .
\\[3mm]
&\qquad\text{ Only the residues in $0,U$ contribute, so...}
\\
J&=2\pi i\cdot 
\frac i{2b}
\cdot[(U+V)+(U-V)]
\\
&=2\pi \frac {-U}b
=2\pi \frac {c-\sqrt{c^2-1}}b
\ .
\end{aligned}
$$
The last expression corresponds to the expected answer from the OP, recalling that $c=a/b$.

Sage check for the partial fraction decomposition:
sage: var('U,z');
sage: V = 1/U
sage: EE = (z^2-1)^2 / z^2 / (z-U) / (z-V)
sage: EE.partial_fraction(z)
-(U^2 - 1)/(U*z - 1) - (U^2 - 1)/((U - z)*U) + (U^2 + 1)/(U*z) + 1/z^2 + 1
sage: bool( _ == (1 + 1/z^2 + (U+V)/z - (V-U)/(z-U) - (U-V)/(z-V) ) )
True

</s>