<s>[INST] Evaluating $\int^{\pi}_0\arctan\left(\frac{p\sin x}{1-p\cos x}\right)\sin(nx) dx$ by differentiation under integral? I saw that 
$$
\int^{\pi}_{0}\arctan \left(\frac{p \sin x}{1-p \cos x}\right) \sin(nx) dx=\frac{\pi}{2n} p^n  
$$
for $$p^2 <1$$
I tried to prove using differentiation under integral but got stuck at this step
$$
I^{\prime} (p)=\int^{\pi}_{0} \frac{\sin x \sin (nx)}{1+p^2-2p \cos x}dx 
$$ 
What to do next?
 [/INST] We assume $p^2<1$. 
First observe that
$$
2I^{\prime} (p)=\!\!\int^{\pi}_{0}\!\! \frac{2\sin x \sin (nx)}{1+p^2-2p \cos x}dx =\!\!\int^{\pi}_{0}\!\! \frac{\cos((n-1)x)}{1+p^2-2p \cos x}dx-\!\!\int^{\pi}_{0}\!\! \frac{\cos((n+1)x)}{1+p^2-2p \cos x}dx. \tag1
$$ Then, by the change of variable $t=\tan (x/2)$, one gets
$$
\int^{\pi}_{0} \frac{1}{1+p^2-2p \cos x}\:dx =\int_0^{\infty} \frac{2 \:dt}{(1-p)^2+(1+p)^2t^2}=\frac{\pi}{1-p^2}.\tag2
$$ From the standard geometric sum identity, one may prove that
$$
\frac{1-p^2}{1+p^2-2p \cos x}=1+2\sum_{k=1}^{n-1}p^k\cos(kx)+\frac{2\:p^n\cos(nx)}{1+p^2-2p \cos x}-\frac{2\:p^{n+1}\cos((n-1)x)}{1+p^2-2p \cos x}.\tag3
$$Since $\displaystyle \int^{\pi}_{0}\cos(kx)dx=0$, for $k=1,2,\cdots$, then integrating $(3)$ from $x=0$ to $x=\pi$ gives, using $(2)$,
$$
\require{cancel}
\cancel{(1-p^2)}\frac{\pi}{\cancel{(1-p^2)}}=\pi+2\:p^n\int^{\pi}_{0}\!\! \frac{\cos(nx)}{1+p^2-2p \cos x}dx-2\:p^{n+1}\int^{\pi}_{0}\!\! \frac{\cos((n-1)x)}{1+p^2-2p \cos x}dx. \tag4
$$ From $(4)$ we deduce
$$
\int^{\pi}_{0}\!\! \frac{\cos(nx)\:dx}{1+p^2-2p \cos x}\!=\!p\!\int^{\pi}_{0}\!\! \frac{\cos((n-1)x)\:dx}{1+p^2-2p \cos x}\!=\cdots=\!p^n\!\int^{\pi}_{0}\!\! \frac{\:dx}{1+p^2-2p \cos x}=\!\frac{\pi\:p^n}{1-p^2}, \tag5
$$ using $(2)$.
Finally, from $(5)$, $(1)$ rewrites
$$
\require{cancel}
2I^{\prime} (p)=\frac{\pi\:p^{n-1}}{1-p^2}-\frac{\pi\:p^{n+1}}{1-p^2}=\frac{\pi\:p^{n-1}}{\cancel{1-p^2}}(\cancel{1-p^2})
$$ or
$$
I^{\prime} (p)=\frac{\pi}2 \:p^{n-1}
$$ then integrating with respect to $p$, using $I(0)=0$, we obtain
$$
I(p)=\frac{\pi}{2n} \:p^n,\quad n\geq1,
$$ that is

$$
\int^{\pi}_{0}\arctan \left(\frac{p \sin x}{1-p \cos x}\right) \sin(nx) \:dx=\frac{\pi}{2n}\: p^n,\qquad p^2<1,\, n\geq1.
$$

</s>