<s>[INST] Integrating $\frac{x^3}{\sqrt{x^2 + 4x + 6}}$ Question:
Evaluate $\displaystyle\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx $.
My attempt:
$\begin{align} \int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx & = \int \frac{x^3}{\sqrt{(x+2)^2 + 2}}\ dx \\& \overset{(1)}= \int \frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)\ }{\sqrt{2\tan^2(t) + 2}}\ dt\\& = \frac{1}{\sqrt{2}}\int\frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)}{\sec(t)}\ dt\\& = \int(\sqrt{2} \tan(t) - 2 )^3 \sec(t)\ dt\\& = \int -8 \sec(t) + 2 \sqrt2 \tan^3(t) \sec(t) - 12\tan^2(t) \sec(t)\\&\qquad + 12 \sqrt2 \tan(t) \sec(t)\ dt\\\\& = -8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  \\
&\qquad + 2\sqrt{2} \int \tan^3(t) \sec(t) \ dt - 12\int\tan^2(t) \sec(t) \ dt\end{align}$
Now both of these integrals can be evaluated using some sort of substitution and integration by parts rule.
This would give us,
$$\boxed{-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C}$$


$(1)$ Here I've made a substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$.


Wolframalpha gives answer as $$\boxed{\frac{1}{3}(x^2 - 5x + 18) \sqrt{x^2 +4x + 6}  - 2 \sinh^{-1}\left(\frac{x+2}{\sqrt{2}}\right) + C}$$
How would I simplify my answer equal to this? Undoing my substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$ is also not an easy task I think.
 [/INST] For indefinite integral of the form
$$I_n=\int \frac{x^n}{\sqrt{x^2 + bx + c}}\ dx 
$$
it is advised that $I_n$ be reduced first to $I_0$ before any substitution. This is achieved by the reduction formula below with $f(x)=x^2+bx+c$
$$\int \frac{f’(x)^{n}}{\sqrt{f(x)}}dx= K_n = \frac2nf’(x)^{n-1}\sqrt{f(x)}+\frac{n-1}n (b^2-4c)K_{n-2}
$$
Thus, apply it to the integral to obtain
\begin{align}
&\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx \\
=&\ \frac18\int \frac{(2x+4)^3-12(2x+4)^2+48(2x+4)-64}{\sqrt{x^2 + 4x + 6}}\ dx \\
=&\ \frac13(x^2-5x+18)\sqrt{x^2 + 4x + 6}-2\int \frac{1}{\sqrt{x^2 + 4x + 6}}\ dx 
\end{align}
</s>