<s>[INST] How can I find the maximum and minimum of $f(x,y)$ in $D=\{ (x,y) \in \Bbb{R}^2 \mid x^2+y^2 \le 4 \}$? How can I find the maximum and minimum of $f(x,y)$ in $D=\{ (x,y) \in \Bbb{R}^2 \mid x^2+y^2 \le 4 \}$
where 
$$f(x,y)=x^4+y^4-x^2-2xy-y^2$$
Answer:
$$f_x=4x^3-2x-2y$$
$$f_y=4y^3-2y-2x$$
Critical points are $(0,0), (1,1), (-1,-1)$. 
All of these critical points are in D.
$$f(0,0)=0$$
$$f(1,1)=-2$$
$$f(-1,-1)=-2$$
Now we can find the extreme values of $f$ on the boundary of the region $x^2+y^2=4$
$$x^2=4-y^2$$
Therefore we have,
$$g(y)=2y^4-8y^2-2y \sqrt{4-y^2}+16$$
$$g'(y)=8y^3-16y-2 \sqrt{4-y^2}+2y^2(4-y^2)^{-1/2}=0$$
$\color{red}{problem:}$
I could do only up to here. I don't know how to solve the above equation to get the values for $y$.
After finding extreme values of this, we can compare those values with the above and we can find the maximum and minimum values.
 [/INST] We have on the boundary $x^2+y^2=4$:
$$f=x^4+y^4-(x^2+y^2)-2xy=(x^2+y^2)^2-2x^2y^2-(x^2+y^2)-2xy=12-2((xy)^2+xy)$$
Note that $|xy| \leq \frac{x^2+y^2}{2}=2$, and the extremal values of $t \mapsto t^2+t$ for $|t| \leq 2$ are $6$ (for $t=2$) and $-\frac{1}{4}$ (for $t=-\frac{1}{2}$).
This shows
$$0 \leq f=12-2((xy)^2+xy) \leq \frac{25}{2}$$
and the values are admitted. $x=y=\pm \sqrt{2}$ yields $f=0$ and $f=\frac{25}{2}$ is admitted for the solutions of $xy=-\frac{1}{2}, x^2+y^2=4$: https://www.wolframalpha.com/input/?i=xy%3D-0.5,+x%5E2%2By%5E2%3D4 (Of course you can compute the solutions by hand, but that is not my job :) ) 
</s>