<s>[INST] How to solve a non-linear system of 3 equations involving $x^2$, $y^2$ and $xy$ only? Let $b_1, b_2,b_3$ be real constants, how to solve the following non-linear system?
\begin{align}
(-3x^2 +  y^2 -6xy) + b_1 &= 0 \qquad\text{(L1)}\\
(\phantom{-} x^2 -3 y^2 -6xy) + b_2 &= 0 \qquad\text{(L2)}\\
(\phantom{.}3x^2 + 3 y^2 -2xy) + b_3 &= 0 \qquad\text{(L3)}
\end{align}

I try (L1)-(L2) which yields: $y^2 = x^2 + \frac{b2-b1}{4}$.
Then, (L2)-3*(L3) leads to this intermediate results:
\begin{align}
x^2 &= \frac{3b_1-2b_2-12b_3}{20}\\
y^2 &= \frac{-2b_1+3b_2-12b_3}{20}
\end{align}
At this point, can I conclude the following statement?
\begin{align}
|x| &= \sqrt{\frac{3b_1-2b_2-12b_3}{20}}\\
|y| &= \sqrt{\frac{-2b_1+3b_2-12b_3}{20}}
\end{align}
What if $(3b_1-2b_2-12b_3) <0$ or $(-2b_1+3b_2-12b_3)<0$?
Does it mean that the system has no solution?

If I plug the previous results of $x^2$ and $y^2$ in (L1), (L2) and (L3), then I obtain:
\begin{align}
xy &= -\frac{1}{40}(3b_1 + 3b_2 + 8b_3)\\
xy &= -\frac{1}{40}(3b_1 + 3b_2 - 52b_3)\\
\end{align}
which is valid only if $b_3 = 0$.
In this case $xy = -\frac{3}{40}(b_1 + b_2)$.
Can I conclude something about the sign of $x$ and $y$?
 [/INST] Since you have three equations and two unknowns, I should define $z=xy$ to make
$$\begin{align}
(-3x^2 +  y^2 -6z) + a &= 0 \tag 1\\
(\phantom{-} x^2 -3 y^2 -6z) + b &= 0 \tag 2\\
(\phantom{.}3x^2 + 3 y^2 -2z) + c &= 0 \tag 3
\end{align}$$
So, four solutions
$$x=\pm\frac{\sqrt{3 a-2 b-3 c}}{2 \sqrt{5}}\qquad y=\pm \frac{\sqrt{3b-2 a-3 c}}{2 \sqrt{5}}\qquad z=\frac{3 a+3 b+2 c}{40}$$
So, as you noticed, if we are in the real domain, this gives already two conditions $3a-2b-3c \geq 0$ and $3b-2a-3c>0$. If these ared fulfilled, then $z=xy$ gives a relation between $a$, $b$ and $c$, but no more.
</s>