<s>[INST] Trigonometric integration 3. $\int \frac{\sqrt{3cos2x-1}}{cosx}$
ATTEMPT:
I did the following substitution 
Let $sinx=t$
$cosxdx=dt$
$I=\frac{\sqrt{3(1-2t^2)-1}}{1-t^2}=\frac{\sqrt{2-6t^2}}{1-t^2}$
Now let $t=\frac{1}{z}$
$dt=-\frac{1}{z^2}dz$
Now $I=\color{red}{-} \frac{\sqrt{2z^2-6}}{z(z^2-1)}$
Lastly substitute $2z^2-6=p^2$
Finally $I=\color{red}{-}2\int \frac{p^2}{(p^2+6)(p^2+4)}dp$ which can be easily be done by using Partial fractions.
Substituting back from $p$ to $z$ to $t$ and finally to $sinx.$
I got $I=2arctan\sqrt{\frac{2-6sin^2x}{4sin^2x}}\color{red}{-}\sqrt{6}arctan\sqrt{\frac{2-6sin^2x}{6sin^2x}}$
But the answer given in the text is $\sqrt{6}arcsin(\sqrt{3}sinx)-2arctan(\frac{2sinx}{\sqrt{3Cos2x-1}})$
Where did i go wrong?
 [/INST] Not sure I can follow your last substitution, but I think the mistake was in dropping a sign when you changed variables from $t$ to $z$.  It appears your answer is off by a sign.
Draw a right triangle and label one degree $\alpha$.Make the side opposite $\alpha$ a length of $\sqrt3\sin x$ and the hypoteneuse a length of $1$.  Then $\sin\alpha=\sqrt3\sin x$ and $\alpha=\sin^{-1}(\sqrt3\sin x)$.  By the Pythagorean theorem, the third side has length $\sqrt{1-3\sin^2x}.$  Note that $\tan\alpha=\frac{\sqrt3\sin x}{\sqrt{1-3\sin^2x}}=\sqrt\frac{3\sin^2x}{1-3\sin^2x}$.  The other acute angle, however has a tangent equal to $\sqrt\frac{1-3\sin^2x}{3\sin^2x}.$  Therefore, $\tan^{-1}\sqrt\frac{1-3\sin^2x}{3\sin^2x}=\frac\pi2-\alpha$.  Similarly, you can find that
$$\tan^{-1}\sqrt\frac{2-6\sin^2x}{4\sin^2x}=\tan^{-1}\frac{\sqrt{3\cos2x-1}}{2\sin x}=\frac\pi2-\tan^{-1}\frac{2\sin x}{\sqrt{3\cos2x-1}}$$
Therefore,
$$\sqrt6\tan^{-1}\sqrt\frac{1-3\sin^2x}{3\sin^2 x}-2\tan^{-1}\sqrt\frac{1-3\sin^2x}{2\sin^2x}=$$
$$\sqrt6(\frac\pi2-\sin^{-1}(\sqrt3\sin x))$$
Still editing...  Why won't this refresh? 
</s>