<s>[INST] How to find $\int_0^\pi (\log(1 - 2a \cos(x) + a^2))^2 \mathrm{d}x, \quad a>1$? Integration by parts is of no success. What else to try?
$$\int_0^\pi (\log(1 - 2a \cos(x) + a^2))^2 \mathrm{d}x, \quad a>1$$
 [/INST] Let $z=e^{ix}$ and then $\cos x=\frac{1}{2}(e^{ix}+e^{-ix})=\frac{1}{2}\left(z+\frac{1}{z}\right),dz=izdx$. We have
\begin{eqnarray*}
\int_0^\pi(\log(1-2a\cos x+a^2))^2dx&=&\frac{1}{2}\int_{-\pi}^{\pi}(\log(1-2a\cos x+a^2))^2dx\\
&=&\frac{1}{2}\int_{|z|=1}\left[\log(1-a\left(z+\frac{1}{z}\right)+a^2)\right]^2\frac{dx}{iz}
\end{eqnarray*}
Note that
\begin{eqnarray*}
\log(1-a\left(z+\frac{1}{z}\right)+a^2)=\log(1-a)-\log z+\log\left(1+\frac{az(1-az)}{1-a}\right)
\end{eqnarray*}
and $\log z$ is analytic in $|z|<1$ and
$$ \log\left(1+\frac{az(1-az)}{1-a}\right)=\sum_{n=1}^\infty\frac{(-1)^{n-1}}{n}\left(\frac{az(1-az)}{1-a}\right)^n. $$
Thus
\begin{eqnarray*}
\int_0^\pi(\log(1-2a\cos x+a^2))^2dx&=&\frac{1}{2i}2\pi i\text{Res}(\left(\log(1-a\left(z+\frac{1}{z}\right)+a^2)\right)^2,0)\\
&=&\pi\log^2(1-a).
\end{eqnarray*}
</s>