<s>[INST] Solve system of 3 equations $x+y+z=0$
$x^2+y^2+z^2=6ab$ 
$x^3+y^3+z^3=3(a^3+b^3)$
this is what i reasoned out so far;
$xyz=a^3+b^3$
$x^2+zx+z^2=3ab$
$y^2+zy+z^2=3ab$
$x^2+xy+y^2=3ab$
$y^2=3ab+zx$
$x^2=3ab+zy$
$z^2=3ab+xy$
I'd prefer a hint rather than a full answer - and how should I solve this kind of systems?
 [/INST] A mechanical technique is offered by "Myself" in the comments. I'll sketch it.
Via Newton's identities we have
$$\begin{array}{ll} x^2+y^2+z^2 & =(x+y+z)^2-2(xy+yz+zx) \\ x^3+y^3+z^3 & =(x+y+z)^3-3(xy+yz+zx)(x+y+z)+3xyz \end{array}$$
(Read the background for information on how these can be obtained.)
Combining with the givens and Vieta's formulas, we have
$$(t-x)(t-y)(t-z)=t^3-3abt-(a^3+b^3).$$
Thus, $x,y,z$ are the roots of this polynomial. If you want to go further and obtain explicit formulas, you can use Cardano's method for the depressed cubic.
</s>