<s>[INST] How to solve the integral $\int\tan^{3}x \sec^{3/2}x\; dx$? How to solve the following indefinite integral 
$$\int \tan^{3}x \sec^{3/2}x \; dx$$
to get the solution in the form of 
$$\large\frac{2}{7}\sec^{7/2}x - \frac{2}{3}\sec^{3/2}x +c$$
I tried taking
$$u = \sec^{2}x \implies du = \tan x \; dx$$ 
$$\large \int \tan^{3} x \sec^{\frac{3}{2}}x\;dx = \int (u-1) u ^{\frac{3}{4}}du = \int u^{\frac{7}{4}}- u^{\frac{3}{4}}du = \frac{4}{11} (\sec x)^{\frac{11}{2}} \frac{4}{7} (\sec x)^{\frac{7}{2}} +c$$
Where did I go wrong?
 [/INST] Your $du$ is wrong. $(\tan x)'=\sec^2 x$ (not the other way around). 
Instead, write the integrand as $\sec x \tan x \tan^2 x \, (\color{maroon}{\sec  x})^{1/2}$ and let $u=\color{maroon}{\sec x}$. Write $\tan^2 x$ in terms of $\color{maroon}{\sec x}$ and note $du$ then is $\sec x\tan x\, dx$.
</s>