<s>[INST] Trig Integration $\displaystyle\int\sec^3(x)\tan^2(x)\,\mathrm{d}x$ $$\int \sec^3\left(x\right)\tan^2\left(x\right)\,\mathrm{d}x$$
Hi, for the question above, I think that substituting $\tan(x)$ would be ok but I couldn't figure out the final step.
$$\int\sec^2(x)\sec(x)\tan^2(x)\,\mathrm{d}x$$
$$\int \sec\left(x\right)u^2(x)\,\mathrm{d}u$$
The answer to this solution is $\sec(x)\tan^3(x)/3$ but it is not correct.
 [/INST] The popular approach is to convert it into integrals of odd powers of secant and then use reduction formulas derived from integration by parts, but there is an alternative.
You can make the substitution $u = sec(x) + tan(x)$.
Note that $1/u = sec(x) - tan(x)$,
because $(sec(x)+tan(x))((sec(x)-tan(x)) = sec^2(x)-tan^2(x) = 1$.
We have:
$$
\begin{align}
\dfrac{u^2 + 1}{2u} &= \dfrac{[(\sec(x) + \tan(x))^2] + [1]}{2[\sec(x) + \tan(x)]} \\
&= \dfrac{[\sec^2(x) + 2\sec(x)\tan(x) + \tan^2(x)] + [\sec^2(x) - \tan^2(x)]}{2[\sec(x) + \tan(x)]}\\
&= \dfrac{[2\sec^2(x) + 2\sec(x)\tan(x)}{2[\sec(x) + \tan(x)]}\\
&= \dfrac{2\sec(x)[\sec(x) + \tan(x)]}{2[\sec(x) + \tan(x)]}\\
&= \sec(x)\\
\end{align}
$$
And also:
$$
\begin{align}
\dfrac{u^2 - 1}{2u} &= \dfrac{[(\sec(x) + \tan(x))^2] - [1]}{2[\sec(x) + \tan(x)]} \\
&= \dfrac{[\sec^2(x) + 2\sec(x)\tan(x) + \tan^2(x)] - [\sec^2(x) - \tan^2(x)]}{2[\sec(x) + \tan(x)]}\\
&= \dfrac{[2\tan^2(x) + 2\sec(x)\tan(x)}{2[\sec(x) + \tan(x)]}\\
&= \dfrac{2\tan(x)[\sec(x) + \tan(x)]}{2[\sec(x) + \tan(x)]}\\
&= \tan(x)\\
\end{align}
$$
And lastly:
$$
\begin{align}
du &= (\sec(x) + \tan(x))' dx \\ 
&= (\sec(x)\tan(x) + \sec^2(x)) dx \\
&= \sec(x)(\sec(x) + \tan(x)) dx \\
&= \left(\dfrac {u^2 + 1}{2u}u\right) dx\\
&= \left(\dfrac {u^2 + 1}{2}\right) dx\\
\end{align}
$$
$$\implies dx =\left(\dfrac { 2 }{u^2 + 1}\right)du$$
So our integral becomes:
$$
\begin{align}
\int \tan^2(x)\sec^3(x) dx &= \int \left(\dfrac{u^2 - 1}{2u}\right)^2 \left(\dfrac{u^2 + 1}{2u}\right)^3 \left(\dfrac { 2 }{u^2 + 1}\right)du \\ 
&= \frac{1}{16} \int \dfrac{(u^2 - 1)^2(u^2 + 1)^2}{u^5}du \\ 
&= \frac{1}{16} \int \dfrac{(u^4 - 1)^2}{u^5}du \\ 
&= \frac{1}{16} \int \dfrac{u^8 - 2u^4 + 1}{u^5}du \\ 
&= \frac{1}{16} \int \left(u^3 - \dfrac{2}{u} + u^{-5}\right) du \\ 
&= \frac{1}{16} \left(\frac{1}{4}u^4 - 2\ln |u| - \frac{1}{4}u^{-4}\right) + C\\ 
&= \frac{1}{64}(u^4 - u^{-4}) - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{64}(u - u^{-1})(u + u^{-1})(u^2 + u^{-2}) - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{16}\left(\frac{u^2 - 1}{2u}\right)\left(\frac{u^2 + 1}{2u}\right)\left(u^2 + \left(\frac{1}{u}\right)^2\right) - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{16}[\tan(x)\sec(x)]\left[(\sec(x)+\tan(x))^2 + (\sec(x)-\tan(x))^2\right] - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{16}[\tan(x)\sec(x)][2\tan^2(x)+2\sec^2(x))] - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{16}[\tan(x)\sec(x)][4\tan^2(x)+2] - \frac{1}{8}\ln |u| + C\\ 
&= \frac{1}{4}tan^3(x)sec(x)+\frac{1}{8}tan(x)sec(x) - \frac{1}{8}\ln |sec(x)+tan(x)| + C\\ 
\end{align}
$$
If you're satisfied with an answer in terms of $u$, that's fine.
But I can see why people use the reduction formula.
</s>