<s>[INST] Study extreme values of functions of several variables. Well, I have to solve the following problem:
$\textit{Study the extreme values of}$ $f(x,y)=x^2+y^2+\alpha x^3y^3$ $\textit{depending on}$ $\alpha$, $where $ $\alpha\in \mathbb{R}$.
It's easy to find that the extreme values of the function must verify that
$$\left\{ 
\begin{array}{ll}
2x+3\alpha x^2y^3 =0\\
2y+3\alpha x^3y^2=0
\end{array}
\right.$$
so this extreme values could be


*

*If $\alpha=0$, only $(0,0)$.

*If $\alpha>0$, $(0,0),(\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}})$.

*If $\alpha<0$, $(0,0),(\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}})$.


Now, we can prove that $(0,0)$ is a relative minimum. Applying the Sylvester theorem, I also proved that if $\alpha>0$, $(\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}})$ are saddle points. My problem is to determine what are $(\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}})$ for $\alpha<0$, whose hessian matrix is semidefinite positive. 
 [/INST] For the case $\alpha<0$, let $t=\sqrt[4]\frac{-2}{3\alpha}$, so the critical points are given by $(0,0), (t,t), \text{ and } (-t,-t)$.
As remarked above, there is a relative minimum at $(0,0)$, 
and there are saddle points at $(t,t)$ and $(-t,-t)$ since
$\;\;\;f_{xx}=2+6\alpha xy^3=2+6\alpha\big(\frac{-2}{3\alpha}\big)=-2$,
$\;\;\;f_{yy}=2+6\alpha x^3y=2+6\alpha\big(\frac{-2}{3\alpha}\big)=-2$,
$\;\;\;f_{xy}=9\alpha x^2y^2=9\alpha\big(\frac{-2}{3\alpha}\big)=-6$, and
$\;\;\;D=f_{xx}f_{yy}-f_{xy}^2=4-36=-32<0$.
</s>