<s>[INST] how to classify critical points for a 2 variable function For each of the following functions, find and classify all critical points.   [That  is,  use  the  second-derivative test to deduce whether each critical point is a local max, a local min, or a saddle.]
$f(x,y) = (x+y)(1-xy)$
my solution:
I first equated gradient of $f$, $\nabla f(a) = 0$
so, $(1-2xy-y^2, 1-2xy-x^2) = (0,0)$
solving two equations I found $y = x,-x$.
and then I found the Hessian matrix
$H =  \begin{bmatrix} 
      -2y & -2x-2y \\
      -2x-2y & -2x \end{bmatrix}$
Now, I am having trouble in how to put in my critical points and check if they are local max, local min or saddle point.
 [/INST] With
$f(x, y) = (x + y)(1 - xy) \tag 1$
and
$\nabla f = (f_x, f_y), \tag 2$
we see that
$f_x = 1 - xy + (x + y)(-y) = 1 - xy - xy - y^2 = 1 - 2xy - y^2, \tag 3$
and likewise
$f_y = 1 - xy + (x  + y)(-x) = 1 - xy - x^2 - xy = 1 - 2xy - x^2; \tag 4$
at critical points of $f(x, y)$, we have
$\nabla f(x, y) = 0, \tag 5$
whence from (3) and (4), at the critical points,
$1 - 2xy - y^2 = 0 = 1 - 2xy - x^2; \tag 6$
we solve this system by observing that it implies
$y^2 = 1 - 2xy = x^2, \tag 7$
so that
$y = \pm x; \tag 8$
using (8) in (6) we may write an equation for $x$:
$0 = x^2 + 2xy - 1 = x^2 \pm 2x^2 - 1; \tag 9$
$x$ must thus obey
$3x^2 - 1 = 0 \tag{10}$
or
$x^2 + 1 = 0; \tag{11}$
we rule out (11) since $x$ is real; thus
$x = \pm \dfrac{1}{\sqrt 3} = \pm \dfrac{\sqrt 3}{3}; \tag{12}$
again from (6),
$y = \dfrac{1 - x^2}{2x} = \dfrac{\dfrac{2}{3}}{2x} = \dfrac{1}{3x}; \tag{13}$
therefore the critical points are
$(x, y) = \left ( \dfrac{\sqrt 3}{3}, \dfrac{\sqrt 3}{3} \right ), \; (x, y) = \left ( -\dfrac{\sqrt 3}{3}, -\dfrac{\sqrt 3}{3} \right ); \tag{14}$
the Hessian $H_f$ of $f(x, y)$ has been provided for us courtesy of our OP kronos:
$H_f = \begin{bmatrix} -2y & -2x-2y \\ -2x-2y & -2x \end{bmatrix}; \tag{15}$
we thus see that
$\det(H_f) = 4xy - 4(x + y)^2 = 4(xy - (x + y)^2) = -4(x^2 +xy + y^2); \tag{16}$
since at the critical points we have
$x = y = \pm \dfrac{\sqrt 3}{3}, \tag{17}$
it follows that at these points
$\det(H_f) = -4, \tag{18}$
which, as is well-known, implies that each critical point is a saddle.
</s>