<s>[INST] How to properly find a critical point for $xy^2$? Let $f(x,y) = xy^2$ and the domain $D = \lbrace (x,y)| x,y\geq0, x^2 + y^2 \leq 3 \rbrace$
$f_x(x,y) = y^2$ and $f_y(x,y) = 2xy$ 
Therefore, the critical points should be $\lbrace (x,y)| y = 0, \sqrt{3} \geq x \geq 0 \rbrace$. 
The determinant of the Hessian is 
$$\det(Hf(x,y))=
\begin{vmatrix}
0 & 2y \\ 
2y & 2x
\end{vmatrix} = 0-4y^2$$
But this doesn't make sense to me because, if $y = 0$, then $f(x,0) = 0$. But on this interval, suppose I chose $(x,y) = (1,\sqrt{2})$, this would be larger and would be the maximum value of the system.  
Why didn't finding the first partials and the Hessian allow me to compute the critical point? 
 [/INST] On the circle of radius $r$, we have $x^2+y^2=r^2$. Therefore,
$$
xy^2=r^2x-x^3\tag1
$$
This implies that the interior critical points are at
$$
(x,y)=\frac r{\sqrt3}\left(\pm1,\pm\sqrt2\right)\tag2
$$
with the corresponding values of
$$
xy^2=\pm r^3\frac2{3\sqrt3}\tag3
$$
At the endponts of $x=\pm r$, we get the values of $xy^2=0$. Therefore,
$$
-r^3\frac2{3\sqrt3}\le xy^2\le r^3\frac2{3\sqrt3}\tag4
$$
Thus, the maxima and minima are on the bounding circle.
If $x^2+y^2=3$, then $r=\sqrt3$ and so $(4)$ says that
$$
-2\le xy^2\le2\tag5
$$
</s>