<s>[INST] Showing $f(x,y):=(y^2-x^2)(y^2-2x^2)$ has no extrema how can I show that $f(x,y):=(y^2-x^2)(y^2-2x^2)$ has no extrema?
I already computed the first derivatives for $x$ and $y$ but that doesn't really help me either
$f_{1_x}=8x^3-6xy^2$
$f_{2_x}=24x^2-6y^2$
$f_{1_y}=-6x^2y+4y^3$
$f_{2_y}=-6x^2+12y^2$
 [/INST] We need to find the critical points by


*

*$f_x=-2x(y^2-2x^2)-4x(y^2-x^2)=0$

*$f_y=2y(y^2-2x^2)+2y(y^2-x^2)=0$
which has solution for $x=y=0$ and for $x,y\neq 0$ we have


*

*$f_x=-2x(y^2-2x^2)-4x(y^2-x^2)=0\\\implies -2(y^2-2x^2)-4(y^2-x^2)=-6y^2+8x^2=0\implies3y^2=4x^2$

*$f_y=2y(y^2-2x^2)+2y(y^2-x^2)=0\implies (y^2-2x^2)+(y^2-x^2)=2y^2-3x^2=0$
which has not solutions.
Then note that
$$f(x,y)=(y^2-x^2)(y^2-2x^2)=y^4-3x^2y^2+2x^4=(y^2-\frac32x^2)^2-\frac14x^4$$
and around$(x,y)=(0,0)$ with $f(0,0)=0$ we have that


*

*$f(0,y)=y^4\ge 0$ 

*$f(x,\sqrt{\frac32}\,x)=-\frac14x^4\le 0$ 
</s>