<s>[INST] Constrained (and non) extrema of $f(x,y)=2x^2-2xy^3+3y^2$ I need to find the critical points on the boundary,inside $D$,outside $D$, and find the image of this function (constrained on $D$).
$f(x,y)=2x^2-2xy^3+3y^2$
$D=\{2x^2+3y^2\le 9\}$
Critical points non-constrained: 
$f_x=4x-2y^3=0$
$f_y=-6xy^2+6y=0$ --> $6y(-xy+1)=0$
$f_y$ is equal to $0$ if $y=0$ or $x=\frac{1}{y}$, plugging it in $f_x$ brings to these critical points : $(0,0),(\pm \frac{1}{2^{\frac{1}{4}}},\pm 2^{\frac{1}{4}})$
using the second derivate test: $(0,0),(\pm \frac{1}{2^{\frac{1}{4}}},\pm 2^{\frac{1}{4}})$ seem to be saddle points (?).
Constrained Critical points (extremas):
$$
\left\{ 
\begin{array}{c}
4x-2y^3=4x\lambda\\
-6xy^2+6y=6y\lambda\\
2x^2+3y^2-9=0
\end{array}
\right. 
$$
$-6xy^2+6y=6y\lambda$ --> $6y(-x+1-\lambda)=0$ from this one I can see that it nullifies when $y=0$ or $x=1-\lambda$. 
If $y=0$ the first equation  gets to $4x(1-\lambda)=0$ which means that it nullifies when $x=0$ or $\lambda = 1$. Looking at the third equation tells us that $(x,y)=(0,0)$ can't be used; 
Plugging $y=0$ in the third equation : possible critical points $(\pm \frac{3}{\sqrt(2)},0)$ 
What I can say about $(\pm \frac{3}{\sqrt(2)},0)$ ?
I think I missed some points.
 [/INST] By AM-GM
$$2x^2+3y^2-2xy^3\leq9+|2xy^3|=9+\frac{1}{\sqrt2}\sqrt{2x^2(y^2)^3}\leq$$
$$\leq9+\frac{1}{\sqrt2}\sqrt{\left(\frac{2x^2+3y^2}{4}\right)^4}\leq9+\frac{1}{\sqrt2}\sqrt{\left(\frac{9}{4}\right)^4}=9+\frac{81}{16\sqrt2}.$$
The equality occurs for $2x^2=y^2$, $-xy^3=|xy^3|$ and $2x^2+3y^2=9,$ which gives
$$(x,y)=\left(-\frac{3}{2\sqrt2},\frac{3}{2}\right),$$
which says that we got a maximal value.
The minimal value is $0$, of course.
In your solution it should be $y=0$, which does not give something with $2x^2+3y^2=9,$ or
$$1-xy=\lambda.$$
</s>