<s>[INST] Integrating $\int \frac{dx}{\sin^2 x \cos^2x-6\sin x\cos x}$ Our goal is to find
$$\int \frac{dx}{\sin^2 x \cos^2x-6\sin x\cos x}$$
Here is my approach:
We divide both numerator and denominator by $\cos^2x$ and the letting $\tan x=u$,our integrand becomes $-\int \frac{u^2+1}{u(6u^2-u+6)} du$. It is do-able but it seems to need partial fractions,but that would be really tiresome and tedious. Is there any clever way by using further substitution to finish from here? Or do we need to change our approach entirely for a nicer solution?
 [/INST] Using you substitution :
Performing a partial fraction decomposition we'll have :
$$F(u) = \frac{a_1}{u} +\frac{a_2u+a_3}{6u^2-u+6}$$
Where :
$$a_1 = uF(u)\Big\vert_{u=0} = \frac{1}{6}$$
Let's calculate $F(1)$ and $F(-1)$ :
$$\begin{cases} F(1)&=\frac{1}{6} +\frac{a_2+a_3}{11} &= \frac{2}{11} \\ F(-1)&=-\frac{1}{6}+\frac{-a_2+a_3}{13} &=-\frac{2}{13} \end{cases} \Longleftrightarrow \begin{cases} a_2+a_3 &=\frac{1}{6} \\ -a_2+a_3&=\frac{1}{6} \end{cases} \Longleftrightarrow \begin{cases}a_2&=0\\ a_3&=\frac{1}{6} \end{cases} $$
Hence you'll get :
$$\frac{u^2+1}{u(6u^2-u+6)} = \frac{1}{6u} +\frac{1}{6(6u^2-u+6)}$$
Now I think the only "messy" part is the second term, so let's antidifferentiate it (dunno if this expression exists lol)  :
$$\frac{1}{6} \int \frac{\mathrm{d}u}{6u^2-u+6} =\frac{1}{6}\int \frac{\mathrm{d}u}{\left(\sqrt{6} u -\frac{1}{2\sqrt{6}}\right)^2+\frac{143}{24}} \qquad \text{Complete the square} $$
Now let $$t=\frac{12u-1}{\sqrt{143}} \Longleftrightarrow \mathrm{d}u = \frac{\sqrt{143}}{12}\mathrm{d}t $$ Therefore :
\begin{align} \frac{1}{6} \int \frac{\sqrt{143}}{12\left(\frac{143}{24}t^2+\frac{143}{24}\right)}\mathrm{d}t&= \frac{1}{3\sqrt{143}}\int \frac{\mathrm{d}t}{t^2+1}\\ &=\frac{1}{3\sqrt{143}} \arctan(t) \\ &=\frac{1}{3\sqrt{143}} \arctan\left(  \frac{12u-1}{\sqrt{143}} \right)\end{align}
Plugging this result to what we got initially :
\begin{align}
-\int \frac{u^2+1}{u(6u^2-u+6)} \mathrm{d}u &= -\frac{1}{3\sqrt{143}} \arctan\left(  \frac{12u-1}{\sqrt{143}} \right)+\frac{1}{6}\int \frac{\mathrm{d}u}{u}  \\ &=-\frac{1}{3\sqrt{143}} \arctan\left(  \frac{12u-1}{\sqrt{143}} \right)-\frac{1}{6}\ln(u) \\ 
&=-\frac{1}{3\sqrt{143}} \arctan\left(  \frac{12\tan(x)-1}{\sqrt{143}} \right)-\frac{1}{6}\ln(\tan(x))\end{align}
Whoops I almost forget the costant :
$$\int \frac{\mathrm{d}x}{\sin^2 x \cos^2x-6\sin x\cos x}=-\frac{1}{6}\ln(\tan(x))-\frac{1}{3\sqrt{143}} \arctan\left(  \frac{12\tan(x)-1}{\sqrt{143}} \right)+\text{C}$$
</s>