<s>[INST] Integrating $\frac{x^3}{\sqrt{x^2 + 4x + 6}}$ Question:
Evaluate $\displaystyle\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx $.
My attempt:
$\begin{align} \int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx & = \int \frac{x^3}{\sqrt{(x+2)^2 + 2}}\ dx \\& \overset{(1)}= \int \frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)\ }{\sqrt{2\tan^2(t) + 2}}\ dt\\& = \frac{1}{\sqrt{2}}\int\frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)}{\sec(t)}\ dt\\& = \int(\sqrt{2} \tan(t) - 2 )^3 \sec(t)\ dt\\& = \int -8 \sec(t) + 2 \sqrt2 \tan^3(t) \sec(t) - 12\tan^2(t) \sec(t)\\&\qquad + 12 \sqrt2 \tan(t) \sec(t)\ dt\\\\& = -8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  \\
&\qquad + 2\sqrt{2} \int \tan^3(t) \sec(t) \ dt - 12\int\tan^2(t) \sec(t) \ dt\end{align}$
Now both of these integrals can be evaluated using some sort of substitution and integration by parts rule.
This would give us,
$$\boxed{-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C}$$


$(1)$ Here I've made a substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$.


Wolframalpha gives answer as $$\boxed{\frac{1}{3}(x^2 - 5x + 18) \sqrt{x^2 +4x + 6}  - 2 \sinh^{-1}\left(\frac{x+2}{\sqrt{2}}\right) + C}$$
How would I simplify my answer equal to this? Undoing my substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$ is also not an easy task I think.
 [/INST] Undoing the substitution is exactly how you would match the two results.
By the Pythagorean theorem, a right triangle with a reference angle $t$ such that $\tan(t)=\frac{x+2}{\sqrt2}$ has its sides occurring in a ratio of $\sqrt2$ (leg adjacent to $t$) to $x+2$ (leg opposite $t$) to $\sqrt{(x+2)^2+2}=\sqrt{x^2+4x+6}$ (hypotenuse). It follows that
$$t = \tan^{-1}\left(\frac{x+2}{\sqrt2}\right) \\
\implies 
\begin{cases} \tan(t) = \frac{x+2}{\sqrt2} \\ 
\sec(t)=\frac{\sqrt{x^2+4x+6}}{\sqrt2} \\ 
\sec(t)+\tan(t) = \frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2} \\
\sec(t)\tan(t) = \frac{(x+2)\sqrt{x^2+4x+6}}2 \\
\ln\left|\sec(t)+\tan(t)\right| = \ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right)
\end{cases}$$
After making the replacements, your antiderivative reduces to
$$-2 \ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right) + (4-3x) \sqrt{x^2+4x+6} + \frac13 (x^2+4x+6)^{3/2} + C$$
Factorize the last two terms as
$$\bigg(4-3x + \frac13 (x^2+4x+6)\bigg) \sqrt{x^2+4x+6} = -\frac13 (x^2+5x-18) \sqrt{x^2+4x+6}$$
Finally, the logarithm can be rewritten using the definition of the inverse hyp. sine:
$$\begin{align*}
\ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right) &= \ln\left(\sqrt{\frac{(x+2)^2+2}2} + \frac{x+2}{\sqrt2}\right) \\
&= \ln\left(\sqrt{\left(\frac{x+2}{\sqrt2}\right)^2+1} + \frac{x+2}{\sqrt2}\right) \\[1ex]
&= \sinh^{-1}\left(\frac{x+2}{\sqrt2}\right)
\end{align*}$$
</s>