<s>[INST] Integration of $ \int x^{2} \sqrt{2x-6} dx $ 
$$ \int x^{2} \sqrt{2x-6} dx = ?$$

My Attempt:
by partial integration
$$ \int x^{2} \sqrt{2x-6} dx = \frac{x^{2} (2x-6)^{3/2}}{3}- \frac{2}{3} \int x(2x-6)^{3/2}dx$$
continuing partial integration
$$ = \frac{x^{2} (2x-6)^{3/2}}{3}- \frac{2}{3} \left[ \frac{x(2x-6)^{5/2}}{5} - \int \frac{(2x-6)^{5/2}}{5} dx\right]  $$
$$ = (x^{2}/3)(2x-6)^{3/2} - (2x/15)(2x-6)^{5/2} + (2/105)(2x-6)^{7/2} + C$$
Is this the correct and best/simplest answer? 
Strangely, the multiple choices only include answers in the form:
$$ A(2x+6)^{7/2} + B(2x+6)^{5/2} + C(2x+6)^{3/2} + D$$
where $A,B,C,D$ are constants.
 [/INST] Your result is correct as soon as you change a sign (typo?):
the result can be written as
$$A(2x-6)^{7/2} + B(2x-6)^{5/2} + C(2x-6)^{3/2} + D$$
Note that
$$\frac{2x}{15}(2x-6)^{5/2}=\frac{2x-6+6}{15}(2x-6)^{5/2}=\frac{1}{15}(2x-6)^{7/2}+\frac{2}{5}(2x-6)^{5/2}.$$
Similarly, you can write  $\frac{x^2}{3}(2x−6)^{3/2}$ as a linear combination of $(2x-6)^{7/2}$, $(2x-6)^{5/2}$, and $(2x-6)^{3/2}$: 
$$\frac{x^2}{3}(2x−6)^{3/2}=\frac{(2x-6)^2/4+3(2x-6)+9}{3}(2x−6)^{3/2}
\\=\frac{1}{12}(2x−6)^{7/2}+(2x−6)^{5/2}+3(2x−6)^{3/2}.$$
Hence, your result can be written as
$$\underbrace{\left(\frac{1}{12}-\frac{1}{15}+\frac{2}{105}\right)}_{1/28}(2x-6)^{7/2} + \underbrace{\left(1-\frac{2}{5}\right)}_{3/5}(2x-6)^{5/2} + 3(2x-6)^{3/2} + C.$$
However, the final result in that form can be obtained more easily by applying the substitution $t=\sqrt{2x-6}$. Then $x=\frac{t^2}{2}+3$, $dx=tdt$ and
$$\int x^{2} \sqrt{2x-6} dx =\frac{1}{4}\int (t^2+6)^2 t (t dt)=\int \left(\frac{t^6}{4}+3t^4+9t^2\right) dt\\
=\frac{t^7}{28}+\frac{3t^5}{5}+3t^3+C
=\frac{1}{28}(2x-6)^{7/2} + \frac{3}{5}(2x-6)^{5/2} + 3(2x-6)^{3/2} + C.$$
</s>