<s>[INST] Given a general region, find the double integral bounded between $y = x$ and $y=3x-x^2 $ $$ J  = \iint_R (x^2-xy)\,dx \,dy, $$
Suppose region R is bounded between $y = x$ and $y=3x-x^2 $
My attempt using vertical integration:
$$ \int^{x=2}_{x=0} \int^{y=3x-x^2}_{y=x} \left({x^2-xy}\right)dy\ dx$$
$$\int^2_0 \left[x^2y-x\frac{y^2}{2}\right]^{3x-x^2}_{x}\, dx$$
$$\int^2_0 \frac{-x^5+4x^4-4x^3}{2} \,dx $$
$$\boxed{J = -\frac{8}{15}}$$

My attempt using horizontal integration :
$$ \int^{y=2}_{y=0} \int^{x=y}_{x=3\,\pm \sqrt{9-y}} \left({x^2-xy}\right)dx\ dy$$
For $ x = 3+\sqrt{9-y}$
$$  \int^2_0 \left[\frac{x^3}{3}-\frac{x^2}{2}y\right]^y_{3+\sqrt{9-y} }\,dy$$
For $ x = 3-\sqrt{9-y}$
$$  \int^2_0 \left[\frac{x^3}{3}-\frac{x^2}{2}y\right]^y_{3-\sqrt{9-y} }\,dy$$

My doubts :
1.) How do I set my limit of integration for horizontal integration, if there is $\pm$ to be considered ?
2.) the answer as negative what does that imply in questions related to double integrals?
Could you guys please help
 [/INST] $y=3x-x^2 \implies y = - (x-\frac{3}{2})^2 + \frac{9}{4}$
So equation of parabola is $ \ 9 - 4y = (2x-3)^2 \ $ and its vertex is $\left(\frac{3}{2}, \frac{9}{4}\right)$.
Intersection of parabola and the line $y = x$ is $(2, 2)$. So you can see that it is below the vertex and to the right of the axis of symmetry.
So if you are doing horizontal integration,
for $0 \leq y \leq 2, $ $x$ is bound between parabola and on the right by the line $y = x$. But for $2 \leq y \leq \frac{9}{4}$, $x$ is bound between parabola at both ends.
Left of the axis of parabola is given by $2x - 3 = - \sqrt{9-4y}$ and right of it is given by $2x - 3 =  \sqrt{9-4y}$

So the integral should be
$\displaystyle \int_0^2 \int_{(3 - \sqrt{9-4y})/2}^y (x^2 - xy) \ dx \ dy \ \ $ +
$\displaystyle \int_2^{9/4} \int_{(3 - \sqrt{9-4y})/2}^{(3 + \sqrt{9-4y})/2} (x^2 - xy) \ dx \ dy \ \ $
$ = - \frac{47}{120} - \frac{17}{120} = - \frac{8}{15}$
which matches your answer using vertical integration.
</s>