<s>[INST] If $x$ and $y$ are real number such that $x^2+2xy-y^2=6$, then find the minimum value of $(x^2+y^2)^2$ If $x$ and $y$ are real number such that $x^2+2xy-y^2=6$, then find the maximum value of $(x^2+y^2)^2$
My attempt is as follows:
$$(x-y)^2\ge 0$$
$$x^2+y^2\ge 2xy$$
$$2(x^2+y^2)\ge x^2+y^2+2xy$$
\begin{equation}
2(x^2+y^2)\ge (x+y)^2\tag{1}
\end{equation}
Solving the given equation:
$$x^2+2xy-y^2=6$$
$$(x+y)^2=2y^2+6$$
So putting the value of $(x+y)^2$ in equation $1$
$$2(x^2+y^2)\ge 2y^2+6$$
$$x^2+y^2\ge y^2+3$$
\begin{equation}
x^2\ge 3\tag{2}
\end{equation}
So $x\in \left(-\infty,-\sqrt{3}\right) \cup \left(\sqrt{3},\infty\right)$
But how to proceed from here?
 [/INST] 
Minimize $z=(x^2+y^2)^2$ s.t. $x^2+2xy-y^2=6$.

The contour curve is a circle with the center at the origin: $x^2+y^2=\sqrt z$.
To minimize $z$ implies to find the smallest radius of the circle. The smallest circle must touch the hyperbola. The slope of tangent to the hyperbola at $(x_0,y_0)$ is:
$$2x+2y+2xy'-2yy'=0 \Rightarrow y'=\frac{x_0+y_0}{y_0-x_0}$$
The normal line to the hyperbola at $(x_0,y_0)$ must pass through the origin:
$$y=\frac{x_0-y_0}{x_0+y_0}x$$ 
The intersection point of the hyperbola and the normal line is $(x_0,y_0)$:
$$\begin{cases}x_0^2+2x_0y_0-y_0^2=6 \\ y_0=\frac{x_0-y_0}{x_0+y_0}\cdot x_0\end{cases} \Rightarrow \begin{cases}x_0^2+2x_0y_0-y_0^2=6 \\ x_0^2-2x_0y_0-y_0^2=0\end{cases} \stackrel{+}\Rightarrow x_0=\sqrt{3+y_0^2} \Rightarrow \\
3+y_0^2+2\sqrt{3+y_0^2}\cdot y_0-y_0^2=6 \Rightarrow y_0^4+3y_0^2-\frac94=0 \Rightarrow \\
y_0^2=\frac{-3+\sqrt{18}}{2},x_0^2=\frac{3+\sqrt{18}}{2} \Rightarrow \\
z(x_0,y_0)=(x_0^2+y_0^2)^2=18 \ \text{(min)}$$
Desmos graph for reference.
</s>