<s>[INST] Solving an analytic geometry problem with euclidean geometry 
$\mathrm{OABC}$ is a tetrahedron with $\overline{\mathrm{OA}}=1$.
There is a point $P$ on $\triangle \mathrm{ABC}$ such that $\cos^2 \alpha+\cos^2 \beta + \cos^2 \gamma = \frac{11}{6}$ where $\alpha=\angle\mathrm{AOP}$, $\beta=\angle\mathrm{BOP}$ and $\gamma=\angle\mathrm{COP}$. 
Describe the trace of such $P$.

We can solve this problem by using the following lemma:

Let's imagine the tetrahedron with four vertices $O(0,0,0)$, $A(a,b,c)$, $B(b,c,a)$ and $C(c,a,b)$.
Since $\overline{\mathrm{OA}}=\overline{\mathrm{OB}}=\overline{\mathrm{OC}}=\overline{\mathrm{AB}}=\overline{\mathrm{BC}}=\overline{\mathrm{CA}}$, we have 
$a^2+b^2+c^2=(a-b)^2+(b-c)^2+(c-a)^2\Rightarrow a^2+b^2+c^2=2(ab+bc+ca)$. 
And since $\vec{\mathrm{OA}}=(a,b,c)$, $\vec{\mathrm{OP}}=(x,y,z)$, we have $\cos^2 \alpha=\left(\frac{\vec{\mathrm{OA}} \cdot \vec{\mathrm{OP}}}{\vert \vec{\mathrm{OA}} \vert \vert\vec{\mathrm{OP}}\vert}\right)^2=\frac{(ax+by+cz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$. 
WLOG $\cos^2 \beta=\frac{(bx+cy+az)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$, $\cos^2 \gamma=\frac{(cx+ay+bz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}$.
Then by the given equation, $\frac{(ax+by+cz)^2+(bx+cy+az)^2+(cx+ay+bz)^2}{(a^2+b^2+c^2)(x^2+y^2+z^2)}=1+\frac{2(ab+bc+ca)(xy+yz+zx)}{(a^2+b^2+c^2)(x^2+y^2+z^2)}=\frac{11}{6}$.
By substituting $a^2+b^2+c^2=2(ab+bc+ca)$, we obtain $5(x^2+y^2+z^2)=6(xy+yz+zx)$.
Now, observe that $P$ is on the plane $x+y+z=a+b+c$. 
So $2(xy+yz+zx)=(x+y+z)^2-(x^2+y^2+z^2)=(a+b+c)^2-(x^2+y^2+z^2)$ and $8(x^2+y^2+z^2)=3(a+b+c)^2$. 
As a conclusion, the trace of $P$ is a circle which is an intersection between the sphere $x^2+y^2+z^2=\frac{3}{8}(a+b+c)^2$ and the plane $x+y+z=a+b+c$.

I think it is a quite simple and nice solution, but I do not want to use such analytic methods.
I tried many times to solve this only with euclidean geometry, but I could not find a better strategy.
Would you help me?
 [/INST] Here's a much shorter answer.
Assuming $OABC$ is a regular tetrahedron of edge length 1.
Place point O at the origin $(0,0,0)$, and let face $ABC$ be parallel to the $xy$ plane, then points $A, B, C$ can be taken as
$A = (\sin \theta , 0, \cos \theta )$
$B = (\sin \theta \cos \phi, \sin \theta \sin \phi, \cos \theta) $
$C = (\sin \theta \cos(- \phi) , \sin \theta \sin(-\phi) , \cos \theta ) $
where $\cos \theta = \sqrt{\dfrac{2}{3}} $ and $\sin \theta = \dfrac{1}{\sqrt{3}} $
and $\phi = \dfrac{2 \pi}{3} $
Hence,
$A = (\dfrac{1}{\sqrt{3}} , 0, \sqrt{\dfrac{2}{3}} )$
$B = (-\dfrac{1}{2 \sqrt{3}}, \dfrac{1}{2}, \sqrt{\dfrac{2}{3}} ) $
$C = (-\dfrac{1}{2 \sqrt{3}} , -\dfrac{1}{2} , \sqrt{\dfrac{2}{3}}) $
Now let $Q$ be a unit vector (lying on the unit sphere that has point $A, B,C$ on it), then $Q$ can be written as
$Q = (\sin \theta \cos \phi, \sin \theta \sin \phi, \cos \theta)$
Since vectors $A, B, C, Q$ all have unit length, it follows that
$\cos^2 \alpha + \cos^2 \beta + \cos^2 \gamma = (A.Q)^2 + (B.Q)^2 + (C.Q)^2 $
In matrix-vector notation, we have,
$ (A.Q)^2 + (B.Q)^2 + (C.Q)^2 = Q^T ( A A^T + B B ^ T + C C^T ) Q $
The $3 \times 3 $ matrix $(A A^T + B B ^ T + C C^T )$ can be evaluated directly, and it comes to:
$(A A^T + B B ^ T + C C^T ) = \begin{bmatrix} \frac{1}{2} && 0 && 0 \\ 0 && \frac{1}{2} && 0 \\ 0 && 0 && 2 \end{bmatrix} $
Hence the condition becomes
$ \dfrac{1}{2} \sin^2 \theta + 2 \cos^2 \theta = \dfrac{11}{6} $
So that,
$ 3 \sin^2 \theta + 12 \cos^2 \theta = 11 $
which becomes,
$ \dfrac{1}{2} ( 15 + 9 \cos 2 \theta ) = 11 $
whose solution is $ \theta = \frac{1}{2} \cos^{-1} \dfrac{7}{9} $
Note that $\cos 2 \theta = \dfrac{7}{9}$ and that $\sin 2 \theta = \dfrac{4 \sqrt{2}}{9} $
The above implies that $Q$ lies on a cone whose semi-vertical angle is $\theta$ as found above.  Therefore, point $P$ lies on a circle whose radius is
$R = \sqrt{\dfrac{2}{3}} \tan \theta = \sqrt{\dfrac{2}{3}} \dfrac{\sin 2 \theta}{1 + \cos 2 \theta} = \sqrt{\dfrac{2}{3}} \dfrac{ 4 \sqrt{2} }{ 16 } = \dfrac{1}{\sqrt{12}}$
</s>