<s>[INST] Extrema of $f(x,y)=(1-x^2-y^2)\cdot xy$ subject to $x^2+y^2\leq 1$ I want to determine the extrema of $f(x,y)=(1-x^2-y^2)\cdot xy=xy-x^3y-xy^3$ subject to $x^2+y^2\leq 1$.
We use Lagrange Multipliers to check the critical points on the circle $x^2+y^2=1$.
To check the critical  points inside the circle, $x^2+y^2<1$, we calculate the critical points of $f(x,y)$.
Let's start with the Lagrange-Multipliers.
We have $f(x,y)=(1-x^2-y^2)\cdot xy=xy-x^3y-xy^3$ und $g(x,y)=x^2+y^2-1=0$.
We consider
\begin{equation*}L(x,y,\lambda )=f(x,y)+\lambda g(x,y)=xy-x^3y-xy^3+\lambda \left (x^2+y^2-1\right )\end{equation*}
We calculate the partial derivatives of $L$.
\begin{align*}&\frac{\partial{L}}{\partial{x}}=y-3x^2y-y^3+2\lambda x \\ &\frac{\partial{L}}{\partial{y}}=x-x^3-3xy^2+2\lambda y \\ &\frac{\partial{L}}{\partial{\lambda }}= x^2+y^2-1 \end{align*}
To calculate the extrema we set each equation equal to zero and solve the system:
\begin{align*}&y-3x^2y-y^3+2\lambda x=0 \ \ \ \ \ \ \ \ (1) \\ & x-x^3-3xy^2+2\lambda y  =0 \ \ \ \ \ \ \ \ (2) \\ &  x^2+y^2-1=0 \ \ \ \ \ \ \ \ (3)\end{align*}
We get the critical points \begin{equation*}P_1\left (\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right ) \ \ , \ \ \ P_2\left (-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}\right )\ \ , \ \ \ P_3\left (-1, 0\right )\ \ , \ \ \ P_4\left (1, 0\right ) \ \ , \ \ \ P_5\left (0, -1\right ) \ \ , \ \ \ \ P_6\left (0, 1\right )\end{equation*}
Now we check inside the circle.
We set the partial derivatives of $f$ equal to zero and solve the system:
\begin{align*}&f_x=0 \Rightarrow y-3x^2y-y^3=0 \ \ \ \ \ \ \ \ (1)\\ &f_y=0 \Rightarrow x-x^3-3xy^2=0\ \ \ \ \ \ \ \ (2)\end{align*}
Then we get the critical points \begin{align*}&Q_1\left (0, 0\right ) \ \ , \ \ \ Q_2\left (\frac{1}{2}, \frac{1}{2}\right )\ \ , \ \ \ Q_3\left (-\frac{1}{2}, -\frac{1}{2}\right )\ \ , \ \ \ Q_4\left (0, -1\right )\ \ , \ \ \ Q_5\left (0, 1\right )\ \ , \ \ \ Q_6\left (-2, -1\right ) \ \ , \ \ \ Q_7\left (0, -1\right )\ \ , \ \\ & \ Q_8\left (-\frac{3}{2}, -\frac{1}{2}\right ) \ \ , \ \ \ Q_9\left (\frac{1}{2}, -\frac{1}{2}\right )\  \ , \ \ \ Q_{10}\left (2, 1\right )\ \ , \ \ \ Q_{11}\left (\frac{3}{2}, \frac{1}{2}\right ) \ \ , \ \ \ Q_{12}\left (-\frac{1}{2}, \frac{1}{2}\right )\end{align*}
Then we have to find the maximal and minimum value of $f$ at all these critical points.
Is everything correct? I am not really sure about the part inside the circle, if all critical points are correct.
 [/INST] The poser of this problem seems to have been having a little fun, as the behavior of this function $ \ f(x,y) \ = \ xy·(1 - x^2 - y^2) \ $ is a bit peculiar.  The factor $ \  1 - x^2 - y^2  \ $ has "four-fold" symmetry about the origin, but the factor $ \ xy \ $ "breaks" this down to just "diagonal" symmetry.  So we may expect extremal points to be found at $ \ (\pm x \ , \ \pm x) \ $ or at $ \ (\pm x \ , \ \mp x) \ \ . $  The level-curves for $ \ f(x,y) \ = \ c \ > \ 0 \ $ lie in the second and fourth quadrants, while those for $ \ c \ < \ 0 \ $ are in the first and third; this holds with one exception we will come to.  (If this seems counter-intuitive, keep in mind that outside the unit circle, $ \ 1 - x^2 - y^2 < 0 \ \ . ) $  For $ \ c \ = \ 0 \ \ , $ the level curve "degenerates" to the union of the two coordinate axes and the unit circle; this marks a "boundary" at which the level-curves change quadrants.
This makes any "critical points" found on the unit circle or the axes degenerate, since all points there correspond to $ \ c \ = \ 0 \ \ . $

What will prove to be of greater interest are the level-curves for $ \ 0 < c < c^{*} \ $ and $ \ -c^{*} < c < 0 \ \ , $ where "loops" appear within the unit disk:  in the first and third quadrants for positive $ \ c \ $ and in the other two quadrants for negative $ \ c \ \ . $  We can find the value of $ \ c^{*} \ $ by considering what happens with $ \ f(x,y) \ $ for $ \ y = \pm \ x \ \ . $  We find that
$$   \ f(x,x) \ = \ x^2·(1 - 2x^2) \ = \ c^{*} \ \ \Rightarrow \ \ 2x^4 - x^2 + c^{*} \ = \ 0 \ \ \Rightarrow \ \ x^2 \ = \ \frac{1 \ \pm \ \sqrt{1 \ - \ 4·2·c^{*}}}{4} \ \ , $$
so $ \ x^2 \ $ has just a single value for $ \ c^{*} = \frac18 \ \ ; $  a similar calculation for $  \ f(x,-x) \ $ gives a symmetrical result.  Hence, the "loops" in the interior of the unit disk are only present for $ \ 0 < c < \frac18 \ \ $ and $ \ -\frac18 < c < 0 \ \ ; $ otherwise, the level-curves are always outside the unit disk for $ \ |c| > \ \frac18 \ \ . $

As for the Lagrange equations $ \ y - y^3 - 3x^2y \ = \ \lambda · 2x \ $ and $ \ x - x^3 - 3xy^2 \ = \ \lambda · 2y \ \ , $ we can solve for $ \ \lambda \ $ to produce
$$ \lambda \ \ = \ \ \frac{y \ - \ y^3 \ - \ 3x^2y}{2x} \ \ = \ \ \frac{x \ - \ x^3 \ - \ 3xy^2}{2y} $$ $$ \Rightarrow \ \ 2x^2 \ - \ 2x^4 \ - \ 6x^2y^2 \ = \ 2x^2 \ - \ 2x^4 \ - \ 6x^2y^2 \ \ \Rightarrow \ \ x^2 · (1 - x^2) \ = \ y^2 · (1 - y^2) \ \ , $$
with this "cross-multiplication" being "safe" since we have already dispensed with points for which $ \ x = 0 \ $ or $ \ y = 0 \ \ . $  The possible pairing of factors in this equation are
$$   x^2 \ = \ y^2 \ \ , \ \  1 - x^2  \ = \ 1 - y^2 \ \ \Rightarrow \ \ y \ = \ \pm \ x  \ \ , $$
$$   x^2 \ = \ 1 - y^2 \ \ , \ \  1 - x^2  \ = \  y^2 \ \ \Rightarrow \ \ x^2 + y^2 \ = \ 1 \ \ . $$
The result we are interested in comes from the critical points you found from $$ \ y·(1 - y^2 - 3x^2) \ = \ 0 \ , \ x·(1 - x^2 - 3y^2) \ = \ 0 \ \ \Rightarrow \ \ 3x^2 + y^2 = 1 \ , \ x^2 + 3y^2 = 1 \ \ , $$
which intersect at the four points $ \ \left( \pm \frac12 \ , \ \pm \frac12 \right) \ . $ [I don't know where your points $ \ Q_6 , Q_8 , Q_{10} \ $ and $ \ Q_{11} \ $ are coming from.]
The points $ \ \left( \ \pm \frac{1}{2} \ , \ \pm \frac{1}{2} \ \right) \ \ [y \ = \ x] \ \ $ correspond to $$ \ f\left( \ \pm \frac{1}{2} \ , \ \pm \frac{1}{2} \ \right) \ = \ \left(\pm \frac{1}{2} \right) · \left(\pm \frac{1}{2} \right) · \left(1 - \frac{1}{4}   - \frac{1}{4} \right) \ \ = \ \frac18 \ \ , $$
making this the absolute maximum of the function in the interior of the unit disk.  (This is the value $ \ c^{*} \ $ at which the loops in the first and third quadrants just "disappear".)  The other set of points $ \ \left( \ \pm \frac{1}{2} \ , \ \mp \frac{1}{2} \ \right) \ \ [y \ = \ -x] \ \ $ correspond to $$ \ f\left( \ \pm \frac{1}{2} \ , \ \mp \frac{1}{2} \ \right) \ = \ \left(\pm \frac{1}{2} \right) · \left(\mp \frac{1}{2} \right) · \left(1 - \frac{1}{4}   - \frac{1}{4} \right) \ \ = \ -\frac18 \ \ , $$
the absolute minmum in the region   (or $ \ -c^{*} \ $ at which the loops in the second and fourth quadrants  vanish.)
</s>