<s>[INST] $\int \frac{dx}{x^3+x^2\sqrt{x^2-1}-x}$ solve:
$$\int \frac{\mathrm dx}{x^3+x^2\sqrt{x^2-1}-x}$$
I tried:
$$\begin{align}\int \frac{\mathrm dx}{x(x^2-1)+x^2\sqrt{x^2-1}}&=\int \frac{\mathrm dx}{x(\sqrt{x^2-1}\sqrt{x^2-1})+x^2\sqrt{x^2-1}}\\&=\int \frac{\mathrm dx}{x\sqrt{x^2-1}(\sqrt{x^2-1}+x)}\end{align}$$
$x=\sin t$
$$\int \frac{\mathrm dt}{\cos t\sin t-\sin^2t}$$
And I can not continue from here.
 [/INST] You can proceed from where you are like this, e.g.
\begin{eqnarray}
\mathcal I &=& \int\frac{dx}{x\sqrt{x^2-1}\left(\sqrt{x^2-1}+x\right)} =\\
&=&-\int\frac{\sqrt{x^2-1}-x}{x\sqrt{x^2-1}}dx=\\
&=&-\int\frac1xdx +\int\frac1{\sqrt{x^2-1}}dx=\\
&=&-\log|x|+\log\left(\sqrt{x^2-1}+x\right)+C
\end{eqnarray}
EDIT
For the second integral use $t=\cosh x$, recalling that $\sinh^2 x= \cosh^2-1$, and $\operatorname{arccosh} x=\log(\sqrt{x^2-1}+x)$. 
EDIT 2
Alternatively, as in comment, $x = \sec t$, brings the second integral to  $\int \sec t dt =\log(\tan t + \sec t) + C= \log(\sqrt{\sec^2t-1}+\sec t)+C\dots$
</s>