<s>[INST] show that the straight lines $(a^2-3b^2)x^2+8abxy+(b^2-3a^2)y^2=0$ form with the lie $ax+by+c=o$ an equilateral triangle show that the straight lines 
$(a^2-3b^2)x^2+8abxy+(b^2-3a^2)y^2=0$ form with the lie ax+by+c=o
an equilateral triangle whose area is $\frac{c^2}{\sqrt{3}(a^2+b^2)}$
is there any other way to solve without intersection points. Also the Angle between the first two lines is 60 degrees.any another way to solve this.
 [/INST] As I was saying, if we have three constants $F,G,H$ and $$ F x^2 + G x y + H y^2 = 0,  $$ we divide through by $x^2$ to arrive at $$ F  + G \frac{y}{x} + H \frac{y^2}{x^2} = 0.  $$ Or, as I said, using the traditional $m = \frac{y}{x},$ we get
$$ F + G m + H m^2 = 0,$$ or in order
$$ H m^2 + G m + F = 0.$$ 
We suspect we are describing two lines passing through the origin. If so, the slope(s) $m$ satisfy
$$ m = \frac{-G \pm \sqrt {G^2 - 4 HF}}{2H}.  $$
So, if $G^2 - 4 F H > 0,$ we do have two different slopes and two lines. For your original problem $$ F = a^2 - 3 b^2, G = 8 a b, H = b^2 - 3 a^2   $$ 
</s>