<s>[INST] Integrating $\frac{x^3}{\sqrt{x^2 + 4x + 6}}$ Question:
Evaluate $\displaystyle\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx $.
My attempt:
$\begin{align} \int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx & = \int \frac{x^3}{\sqrt{(x+2)^2 + 2}}\ dx \\& \overset{(1)}= \int \frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)\ }{\sqrt{2\tan^2(t) + 2}}\ dt\\& = \frac{1}{\sqrt{2}}\int\frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)}{\sec(t)}\ dt\\& = \int(\sqrt{2} \tan(t) - 2 )^3 \sec(t)\ dt\\& = \int -8 \sec(t) + 2 \sqrt2 \tan^3(t) \sec(t) - 12\tan^2(t) \sec(t)\\&\qquad + 12 \sqrt2 \tan(t) \sec(t)\ dt\\\\& = -8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  \\
&\qquad + 2\sqrt{2} \int \tan^3(t) \sec(t) \ dt - 12\int\tan^2(t) \sec(t) \ dt\end{align}$
Now both of these integrals can be evaluated using some sort of substitution and integration by parts rule.
This would give us,
$$\boxed{-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C}$$


$(1)$ Here I've made a substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$.


Wolframalpha gives answer as $$\boxed{\frac{1}{3}(x^2 - 5x + 18) \sqrt{x^2 +4x + 6}  - 2 \sinh^{-1}\left(\frac{x+2}{\sqrt{2}}\right) + C}$$
How would I simplify my answer equal to this? Undoing my substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$ is also not an easy task I think.
 [/INST] By hyperbolic substitution
Letting $x+2=\sqrt 2 \sinh \theta$ transforms the integral into
$$
\begin{aligned}
I= & \int \frac{(\sqrt{2} \sinh \theta-2)^3}{\sqrt{2} \cosh \theta} \cdot \sqrt{2} \cosh \theta d \theta \\
= & \int\left(2 \sqrt{2} \sinh ^3 \theta-12 \sinh ^2 \theta+12 \sqrt{2} \sinh \theta-8\right) d \theta \\
= & 2 \sqrt{2} \int\left(\cosh ^2 \theta-1\right) d(\cosh \theta)-12 \int \frac{\cosh 2 \theta-1}{2} d \theta +12 \sqrt{2} \cosh \theta-8 \theta \\
= &\frac{2 \sqrt{2}\cosh ^3 \theta}{3}-3\sinh 2\theta +10\sqrt{2} \cosh \theta-2 \theta+C \cdots (*)\\=& \frac{2 \sqrt{2}}{3} \cosh \theta\left(15+\cosh ^2 \theta-\frac{9}{\sqrt{2}} \sinh \theta\right)-2 \theta+C
\end{aligned}
$$
Putting back $x+2=\sqrt 2 \sinh \theta$ yields
$$
\begin{aligned}
I & =\frac{2 \sqrt{2}}{3} \sqrt{1+\left(\frac{x+2}{\sqrt{2}}\right)^2}\left[15+1+\left(\frac{x+2}{\sqrt{2}}\right)^2-\frac{9}{\sqrt{2}} \cdot \frac{x+2}{\sqrt{2}}\right] +C\\
& =\frac{1}{3} \sqrt{x^2+4 x+6}\left(x^2-5 x+18\right)-2 \sinh ^{-1}\left(\frac{x+2}{\sqrt{2}}\right)+C
\end{aligned}
$$
By trigonometric substitution
Questioner using $x+2=\sqrt 2\tan t$ got the answer
$$I=-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C$$
Using $x+2=\sqrt2\tan t =\sqrt 2\sinh \theta \Leftrightarrow \tan t=\sinh \theta \textrm{ and } \sec t=\cosh \theta$, we get
$$
\begin{aligned}
I=-2 & \ln |\sec t+\tan t|+\frac{2 \sqrt{2}}{3} \sec ^3 t+10 \sqrt{2} \sec t -6 \sec t \tan t+C \\
=- & 2 \ln |\cosh \theta+\sinh \theta|+\frac{2 \sqrt{2}}{3} \cosh ^3 \theta+10\sqrt{2} \cosh \theta -6 \cosh \theta \sinh \theta+C
\end{aligned}
$$
Since
$\ln |\cosh \theta+\sinh \theta |=  \ln \left(\frac{e^\theta+e^{-\theta}}{2}+\frac{e^\theta-e^{-\theta}}{2}\right) 
= \theta$, therefore $$I= \frac{2 \sqrt{2}\cosh ^3 \theta}{3}-3\sinh 2\theta +10\sqrt{2} \cosh \theta-2 \theta+C \cdots (*)$$
We can now conclude that both substitutions gives the same result.
</s>