<s>[INST] Evaluating $\int x^2 \sqrt{x^2-1} dx$ How do I evaluate the following indefinite integral?
$$\int x^2 \sqrt{x^2-1} dx$$
Through integration of parts, I have obtained 
$$ \frac{x}{3}(x^2-1)^{3/2} - \frac{1}{3} \int (x^2-1)^{3/2} dx $$
I've attempted evaluating the second term through substitution, where
$$ x =  \sec(u)$$
However, I am stuck with 
$$ \frac{x}{3}(x^2-1)^{3/2} - \frac{1}{3} \int \tan^4(u) \sec (u) du $$
What would be my next step?
 [/INST] Used a different version of the secant reduction formula. I really enjoyed this problem! 
Let $x=\sec(u)$ and $dx=\sec(u)\tan(u)\,dx$ so that:
$$\int x^2\sqrt{x^2-1}\,dx=\int\sec^2(u)\sqrt{\sec^2(u)-1}\sec(u)\tan(u)\,du$$
Now simplify, noting that $\sec^2(u)-1=\tan^2(u)$:
$$=\int\sec^3(u)\tan^2(u)\,du$$
And again, using $\sec^2(u)-1=\tan^2(u)$ and simplifying:
$$\int(\sec^5(u)-\sec^3(u))\,du$$
Apply the version of the secant reduction formula containing tangent, which is:
$$\int\sec^n(u)\,du=\frac{\tan(u)\sec^{n-2}(u)}{n-1}+\frac{n-2}{n-1}\int\sec^{n-2}(u)\,du$$
To get:
$$\int(\sec^5(u)-\sec^3(u))\,du=\frac{\tan(u)\sec^3(u)}{4}+\frac{3}{4}\bigg(\frac{\tan(u)\sec(u)}{2}+\frac{1}{2}\ln|\sec(u)+\tan(u)|\bigg)$$
$$-\bigg(\frac{\tan(u)\sec(u)}{2}+\frac{1}{2}\ln|\sec(u)+\tan(u)|\bigg)$$
Next apply the substitutions $x=\sec^{-1}(u)$ and $\tan(\sec^{-1}(u)=x\sqrt{1-\frac{1}{x^2}}$ and simplify:
$$\int x^2\sqrt{x^2-1}\,dx=\frac{1}{4}x^4\sqrt{1-\frac{1}{x^2}}-\frac{1}{8}x^2\sqrt{1-\frac{1}{x^2}}-\frac{1}{8}\ln\bigg|x+\sqrt{1-\frac{1}{x^2}}\bigg|+C$$
</s>