<s>[INST] Condition when angle between two lines is ${\pi\over 3}$ The whole question is-

Show that if the angle between the lines whose direction cosines are
  given $l+m+n=0$ and $fmn+gnl+hlm=0$ is ${\pi\over 3}$ then ${1\over
 f}+{1\over g}+{1\over h}=0$.

I'm trying to solve the problem in the following manner-
From the first equation $n=-l-m$, substituting this value of $n$ in the second equation we get-
$fm(-l-m)+g(=l-m)l+hlm=0$
$\implies g\left({l\over m}\right)^2+(f+g+h)\left({l\over m}\right)+f=0$
Now, the roots of this equation are ${l_1\over m_1}$ and ${l_2\over m_2}$. So, product of them ${l_1\over m_1}.{l_2\over m_2}={f\over g}\implies {l_1 l_2\over f}={m_1 m_2\over g}$.
Similarly, we get ${m_1 m_2\over g}={n_1 n_2\over h}$.
Hence, ${l_1 l_2\over f}={m_1 m_2\over g}={n_1 n_2\over h}=K$(say)
Thus, $\cos {\pi\over3}=l_1 l_2+m_1 m_2+n_1n_2=K(f+g+h)\implies K=\frac{\sqrt{3}}{2(f+g+h)}$.
Now, I can't proceed further. I can't prove ${1\over f}+{1\over g}+{1\over h}=0$.
Can anybody solve the problem? Thanks for assistance in advance.
 [/INST] In general if a, b, c and d, e, f are components of 2 vectors then the direction cosines are for the first vector :
$\frac{a}{\sqrt{a^2 + b^2 + c^2}}$ and $\frac{b}{\sqrt{a^2 + b^2 + c^2}}$ and $\frac{c}{\sqrt{a^2 + b^2 + c^2}}$ similarly for the second vector.
Then the angle $\theta$ between these vectors is
$$cos\theta = \frac{ad + be + cf}{\sqrt{a^2 + b^2 + c^2}\sqrt{d^2 + e^2 + f^2}}$$
Using the definition of dot product.  In other words $cos\theta$ equals the sum of product of the directional cosines of corresponding vectors.  In this question we have
$$cos\theta = l(fmn) + m(gnl) + n(hlm) = lmn(f + g + h)$$
The next step is to prove this equal 1/2.
From the question we have 
$$l + m + n = 0$$
$$l^2 + m^2 + n^2 = 1$$
Since there are three unknowns but only 2 equations, we can only express l and n in terms of m :
$$\implies l^2 + lm + m^2 = \frac{1}{2}$$
$$\implies l = \frac{-m + D}{2}$$
$$\implies n = \frac{-m - D}{2}$$
Where $D = \sqrt{2 - 3m^2}$. It does not matter which root we take for l the other root will be for n.
Also we have other conditions:
$$fmn + gnl + hlm = 0 ...      (1)$$
$$(fmn)^2 + (gnl)^2 + (hlm)^2 = 1   ...    (2)$$
$$\frac{1}{f} + \frac{1}{g} + \frac{1}{h} = 0  \implies fg + gh + fh = 0... (3)$$
Put l and n into (1) we have
$$-fm(m + D) - g(1 - 2m^2) + hm(-m + D) = 0...(4)$$
Put l and n into (2) we have
$$f^2m^2(2 - 2m^2 + 2mD) + g^2(1 - 2m^2)^2 + h^2m^2(2 - 2m^2 -2mD) = 4...(5)$$
Eliminate f from (3) and (4) we have
$$f^2m(m + D) + fg(2mD - 2m^2 + 1) + g^2(1 - 2m^2) = 0$$
$$\implies f = \frac{2m^2 - 2mD - 1\pm\sqrt{(2mD -2m^2 + 1)^2 - 4(m + D)m(1 - 2m^2)}}{2m(m + D)}g$$
$$\implies f = \frac{2m^2 - 2mD - 1 \pm 1}{2m(m + D)}g$$
We take plus 1 for simplicity since the other root may give a result making $cos\theta$ greater than 1 or less than -1.
$$f = \frac{m - D}{m + D}g$$
Eliminate h from (4) and (5) gives
$$f^2m^2(2 - 2m^2 + 2mD) + g^2(1 - 2m^2)^2 + fgm(m + D)(1 - 2m^2) = 2$$
Put f into this equation we get
$$m^2(m - D)^2g^2 + g^2(1 - 2m^2)^2 + (m - D)mg^2(1 - 2m^2) = 2$$
$$\implies g^2(1 - Dm - m^2) = 2$$
$$\implies g = \sqrt{\frac{2}{1 - Dm - m^2}}$$
$$\implies f = \frac{m - D}{m + D}\sqrt{\frac{2}{1 - Dm - m^2}}$$
$$\implies h = \frac{D - m}{2m}\sqrt{\frac{2}{1 - Dm - m^2}}$$
Hence put l, m, n, f, g, h to the cosine expression we get
$$cos\theta = lmn(f + g +h) = \frac{-(m - D)}{2}m\frac{-(m + D)}{2}\sqrt{\frac{2}{1 - Dm - m^2}}(\frac{m - D}{m + D} + 1 + \frac{D - m}{2m})$$
$$=\frac{m - D}{4}\sqrt{\frac{2}{1 - Dm - m^2}}$$
$$=\frac{1}{4}\sqrt{\frac{2(m^2 + 2 - 3m^2 - 2Dm)}{1 -  Dm - m^2}} = \frac{1}{2}$$
</s>