<s>[INST] Calculating cosine of dihedral angle 
Let $O,A,B,C$ be points in space such that $\angle AOB=60^{\circ},\angle BOC=90^{\circ},\angle COA=120^{\circ}$ Let $\theta$ be the acute angle between the planes $AOB$ and $AOC$. Find $\cos\theta.$

Here is what I did:
I let $AO=OB=BA=1, OC=1$. Then we can compute $BC=\sqrt{2},AC=\sqrt{3}$. From here, I let $A=(0,0,0),B=(1,0,0),C=(\frac{1}{2},\frac{\sqrt{3}}{2},0),C=(1,0,\sqrt{2})$ (I found the coordinates of $C$ by noticing $\angle ABC=90^{\circ}$ since $AB^2+BC^2=AC^2$). Then I found the equation of $AOC$ to be $-\sqrt{6}x+\sqrt{2}y+\sqrt{3}z=0$ and the equation of $AOB$ to be $z=0$, so using the formula here, I got $\cos\theta=\frac{\sqrt{3}}{11}$, but the answer is $\frac{1}{3}$. Where did I mess up?
Also a solution without using coordinates would be very nice to see!
 [/INST] You can also use the Spherical Law of Cosines.
We imagine a sphere centered at $O$, and we take $A$, $B$, $C$ the points where the rays from $O$ meet that sphere as vertices of a curvilinear triangle on the surface of that sphere. The "sides" of that triangle are arcs of great circles, and the lengths of those arcs are directly proportional to the measures of the angles between the rays. (With an appropriate choice of radius, we can say that the lengths are equal to the measures.) The "angles" of the triangle match the dihedral angles between the planes formed by those rays.
Now, since angle $\theta$ is opposite side $\overline{BC}$, we calculate as follows ...

$$\cos\theta = \frac{\cos \angle BOC-\cos \angle AOB \cos \angle AOC}{\sin \angle AOB \sin \angle AOC} = \frac{\cos 90^\circ - \cos 60^\circ \cos 120^\circ}{\sin 60^\circ \sin 120^\circ} \\[20pt]
=\frac{0 - \frac{1}{2}\frac{-1}{2}}{\frac{\sqrt{3}}{2}\frac{\sqrt{3}}{2}} = \frac{1/4}{3/4} = \frac{1}{3}$$  

</s>