<s>[INST] Prove that: $\sin \,\cot^{-1} \cos \,\tan^{-1}x = \sqrt{\dfrac{x^2+1}{x^2+2}} $. Prove that: $\sin \,\cot^{-1} \cos \,\tan^{-1}x = \sqrt{\dfrac{x^2+1}{x^2+2}} $.
This question was taken from Miscellaneous Example of S.L. Loney's Trigonometry.
Since, the question involves 2 parts, LHS and RHS, LHS is completely Trigonometric while, RHS is algebraic, so my approach was to get taylor series for $x+1$ then differentiate it and keep on doing to show that both sides are same... But, I wonder if there is an trigonometric solution for the question too or not!
Please help.
 [/INST] Let $y=\tan^{-1}x$. Then $\tan y=x$ and so drawing a right triangle with angle $y$ and sides $1,x,\sqrt{1+x^2}$, we find $\cos y=\dfrac{1}{\sqrt{1+x^2}}$.
Now let $t=\cot^{-1}(\cos y)$. Then $\cot t=\cos y=\dfrac{1}{\sqrt{1+x^2}}$, so drawing a right triangle with angle $t$, we find the hypotenuse is $\sqrt{2+x^2}$, so $\sin t=\sqrt{\dfrac{1+x^2}{2+x^2}}$.
</s>