<s>[INST] If $x$ and $y$ are real number such that $x^2+2xy-y^2=6$, then find the minimum value of $(x^2+y^2)^2$ If $x$ and $y$ are real number such that $x^2+2xy-y^2=6$, then find the maximum value of $(x^2+y^2)^2$
My attempt is as follows:
$$(x-y)^2\ge 0$$
$$x^2+y^2\ge 2xy$$
$$2(x^2+y^2)\ge x^2+y^2+2xy$$
\begin{equation}
2(x^2+y^2)\ge (x+y)^2\tag{1}
\end{equation}
Solving the given equation:
$$x^2+2xy-y^2=6$$
$$(x+y)^2=2y^2+6$$
So putting the value of $(x+y)^2$ in equation $1$
$$2(x^2+y^2)\ge 2y^2+6$$
$$x^2+y^2\ge y^2+3$$
\begin{equation}
x^2\ge 3\tag{2}
\end{equation}
So $x\in \left(-\infty,-\sqrt{3}\right) \cup \left(\sqrt{3},\infty\right)$
But how to proceed from here?
 [/INST] Play with linear combinations of $x^2+y^2$ and the expression from the given equation. This way we find for $a>1$:
$$a(x^2+y^2) - 6 = (a-1)x^2-2xy+(a+1)y^2=(\sqrt{a-1}x-\sqrt{a+1}y)^2+(2\sqrt{a^2-1}-2)xy$$
We get rid of the last term if we let $a=\sqrt 2$:
$$\tag1\sqrt 2(x^2+y^2) -6 = \left(\sqrt{\sqrt 2-1}\,x-\sqrt{\sqrt 2+1}\,y\right)^2\ge 0.$$
This gives us the lower bound $x^2+y^2\ge 3\sqrt 2$ (and so $(x^2+y^2)^2\ge18$).
Is the inequality $(1)$ sharp, i.e., can equality hold? Clearly, equality holds iff  $x=\sqrt{\sqrt 2+1}\,t$ and $y=\sqrt{\sqrt 2-1}\,t$ for some $t$. But does there actually exist $t$ such that the original condition is met? We compute
$$ x^2+2xy-y^2=(\sqrt2+1)t^2+2t^2+(\sqrt2-1)t^2=\sqrt 2t^2$$
and this is indeed $=6$ for suitable $t$ (namely $t=\sqrt{3\sqrt 2}$).
</s>