<s>[INST] By application of calculus of residues, can you please solve this problem? By application of calculus of residues, prove that
$$
\int_{0}^{2\pi}
\frac{\cos^{3}\left(3\theta\right)}
{1 - 2p\cos\left(2\theta\right) + p^{2}}
\,\mathrm{d}\theta = \frac{\pi\left(1 - p + p^{2}\right)}{1 - p}
$$
I have attempted the above question using the substitution
$z = \mathrm{e}^{\mathrm{i} \theta}$ and I have also used the substitution
$\cos\left(x\right) =
\mathrm{e}^{\mathrm{i}\theta}$ but I didn't get the solution correctly.
Solution
\begin{equation*}
\text{Let}\, z=e^{i\theta}\, \hspace{2mm} dz=ie^id\theta,\,\hspace{2mm} dz=izd\theta\\
\end{equation*}
\begin{equation*}
\cos\theta=\frac{z+z^{-1}}{2}\\ 
\end{equation*}
\begin{equation*}
\cos2\theta=\frac{z^2+z^{-2}}{2}\\ 
\end{equation*}
\begin{equation*}
\cos3\theta=\frac{z^3+z^{-3}}{2}\\ 
\end{equation*}
\begin{equation*}
 \begin{split}
 \int_0^{2\pi} \frac{\cos ^3{3\theta}d\theta}{1-2p\cos 2\theta+p^2} &= \oint\frac{(\frac{z^3+z^{-3}}{2})^3}{1-2p(\frac{z^2+z^{-2}}{2})+p^2}\cdot\frac{dz}{iz}\\
 &= \frac{1}{8}\oint\frac{-(z^3+z^{-3})^3i}{[z-p(z^3+z^{-1})+p^2z]}\cdot dz\\
 &= \frac{1}{8}\oint\frac{(z^3+1)^3i}{z^8[pz^4+(p^2+1)z^2-p]}\cdot dz
 \end{split}
\end{equation*}
There is a pole at z=0 of order 8.
Kindly help, to determine the remaining poles and the residues.
 [/INST] $\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\mc}[1]{\mathcal{#1}}
 \newcommand{\mrm}[1]{\mathrm{#1}}
 \newcommand{\on}[1]{\operatorname{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$
$\ds{\bbox[5px,#ffd]{\int_{0}^{2\pi}
{\cos^{3}\pars{3\theta} \over
1 - 2p\cos\pars{2\theta} + p^{2}}
\,\dd\theta = \require{cancel}
\cancelto{0}{{\pi\pars{1 - p + p^{2}} \over
1 - p}}}:\ {\Large ?}}$.

\begin{align}
&\bbox[5px,#ffd]{\int_{0}^{2\pi}
{\cos^{3}\pars{3\theta} \over
1 - 2p\cos\pars{2\theta} + p^{2}}\,\dd\theta}
\\[5mm] = &\
-\int_{-\pi}^{\pi}
{\cos^{3}\pars{3\theta} \over
1 - 2p\cos\pars{2\theta} + p^{2}}\,\dd\theta
\\[5mm] = &\
-2\int_{0}^{\pi}
{\cos^{3}\pars{3\theta} \over
1 - 2p\cos\pars{2\theta} + p^{2}}\,\dd\theta
\\[5mm] = &\
2\int_{-\pi/2}^{\pi/2}
{\sin^{3}\pars{3\theta} \over
1 + 2p\cos\pars{2\theta} + p^{2}}\,\dd\theta =
\bbx{\large 0} \\ &
\end{align}

*

*because the integrand is an $\ds{\underline{odd\ function}}$

*and the integral is evaluated between
symmetric limits $\ds{\pars{~\mbox{i.e.}\ \pm{\pi \over 2}~}}$.

</s>