<s>[INST] Triangle-geometry problem Here is the question:

$\cos(A-B) = \frac{7}{8}$, $\cos(C) = ?$
By the Law of Cosines, I get:
$AB^2 = 41-40\cos(C)$
I also tried to expand $\cos(A-B)$ by the compound angle formula, getting:

*

*$\cos(A)\cos(B) + \sin(A)\sin(B)$
Which by the Law of Sines becomes:

*

*$\cos(A)\cos(B) + \frac{5}{4} \sin(B)^2 = \frac{7}{8}$
That's where I have been able to get so far. One thing though that has been bothering me is whether $AB =3$. I am tempted to go down that way because of the Pythagorean triple $3^2 + 4^2 = 5^2$. However, they have not specified that $\angle{A} = \frac{\pi}{2}$, so I am worried about wrongly assuming it. Any assistance would be greatly appreciated.
 [/INST] You can write the angles $A$ and $B$ as follows
$ A = \left( \dfrac{A + B}{2} \right) + \left( \dfrac{A - B}{2} \right) $
and
$ B = \left( \dfrac{A + B}{2} \right) - \left( \dfrac{A - B}{2} \right ) $
The angle $\theta = \dfrac{A - B}{2}$ is known, it is
$\theta = \dfrac{1}{2} \cos^{-1} (\frac{7}{8} ) $
From the law of sines we have
$ \dfrac{\sin A}{\sin B} = \frac{5}{4} $
Hence, since $C = \pi - (A+B) $  then $\phi =  \dfrac{A + B}{2} = \dfrac{\pi - C}{2} $
$ \dfrac{\sin( \phi + \theta ) }{\sin (\phi - \theta) } = \dfrac{5}{4} $
The only unknown here is $\phi$.
Cross multiplying and expanding
$ 4 \sin (\phi + \theta ) = 5 \sin(\phi - \theta) $
$ 4 ( \sin(\phi) \cos(\theta) + \cos(\phi) \sin(\theta) ) = 5 (\sin(\phi) \cos(\theta) - \cos(\phi) \sin(\theta) )$
Collecting terms
$ \sin(\phi) \cos(\theta) = 9 \cos(\phi) \sin(\theta) $
Hence,
$ \tan(\phi) = 9 \tan(\theta) $
Now
$\theta = \frac{1}{2} \cos^{-1} \frac{7}{8} $
Therefore,
$\cos(\theta) = \sqrt{ \dfrac{1 + \frac{7}{8} }{2} } = \dfrac{\sqrt{15}}{4} $
and
$\tan(\theta) = \sqrt{ \frac{16}{15} - 1 } = \dfrac{1}{\sqrt{15}} $
Hence
$\tan(\phi) = \dfrac{9}{\sqrt{15}} $
from which
$\sec(\phi) = \sqrt{\frac{32}{5}} $
and
$\cos(\phi) = \sqrt{\frac{5}{32}} $
Thus
$\cos(2 \phi) = 2 \left( \frac{5}{32} \right) - 1 = - \dfrac{11}{16}  $
But $ 2 \phi = \pi - C $, therefore, $C = \pi - 2 \phi $
From which $\cos(C) = - \cos(2 \phi) = \boxed{\dfrac{11}{16}}$
</s>