<s>[INST] If the points $(0,0),(a,11),(b,37)$ are vertices of equilateral triangle what is the value of ab? If the points $(0,0),(a,11),(b,37)$ are vertices of equilateral triangle, find the value of $ab$
 [/INST] Pretty lengthy equations (and probably bad method to solve). So here are the equations :
$$a^2+11^2=b^2+37^2=(b-a)^2+(37-11)^2$$
$$\implies b^2+a^2-2ab +676 = a^2+121\implies b^2-2ab+555=0\,\,\,\,(1)$$
$$b^2 = a^2-1278 \implies b=\sqrt{a^2-1278}\,\,\,\,\,\,(2)$$
substitute $b$ in $(1)$$$\implies (a^2-1278)+555 = 2a(\sqrt{a^2-1278})$$
Now this is ugly step :- square both sides=>
$$\implies a^4 + 693^2 -2(693)a^2 = 4a^2(a^2-1278)$$
And if you assign $a^2$ to some $t$ it is in quadraric form and should be solvable. 
</s>