<s>[INST] How to find the number of integral solutions of a given equation analytically? I was given $$\tan^{-1} x+\cot^{-1} y= \tan^{-1} 3$$
and asked to find the number of integral solutions.
I was able to find two, which are $(1,2) ; (2,7)$ which was obtained by solving the given equation by taking the tangent of  both sides and writing $\cot y \\$ as  $  \tan \frac{1}{y}$
after which I used trial and error.
However, this doesn't avoid the possibility of missing solutions.
What's the analytical method of solving such problems?
I have come across this question here  on solving equations using Simon's favourite factoring technique but, I don't really get that. Hence I'm asking here.
 [/INST] According to your description of the problem, you got at least this:
$$
\frac{x+y^{-1}}{1-xy^{-1}} = 3
$$
Multiply numerator and denominator of the left side with $y$:
$$
\frac{xy+1}{y-x} = 3
$$
which means
$$
xy+1 = 3y-3x
$$
or
$$
(x-3)(y+3)=-10
$$
Now you only have to check in how may ways $-10$ can be expressed as product of two numbers. Then you get the candidates of the possible solutions.
$$
(x-3,y+3) \in\{(-10,1),(-5,2),(-2,5),(-1,10),(1,-10),(2,-5),(5,-2),(10,-1)\}
$$
That is
$$
(x,y) \in\{(-7,-2),(-2,-1),(1,2),(2,7),(4,-13),(5,-8),(8,-5),(13,-4)\}
$$
However, we have only shown that this is a necessary condition. We also have to account for the fact that we would have got the same necessary condition if we added an integer multiple of $\pi$ to one of the sides of the original equation. So we can expect that some of the candidates do not fulfill the original equation, but make the left side and the right side differ by an integer multiple of $\pi.$
If we use the convention $\tan^{-1}x\in \left(-\frac{\pi}{2},\frac{\pi}{2}\right)$ and $\cot^{-1}y\in \left(-\frac{\pi}{2},\frac{\pi}{2}\right]\setminus\{0\}$, we get
\begin{eqnarray}
\tan^{-1}(-7)+\cot^{-1}(-2) &=& \tan^{-1}(3) -\pi \\
\tan^{-1}(-2)+\cot^{-1}(-1) &=& \tan^{-1}(3) -\pi \\
\tan^{-1}(1)+\cot^{-1}(2) &=& \tan^{-1}(3)  \\
\tan^{-1}(2)+\cot^{-1}(7) &=& \tan^{-1}(3)  \\
\tan^{-1}(4)+\cot^{-1}(-13) &=& \tan^{-1}(3) \\
\tan^{-1}(5)+\cot^{-1}(-8) &=& \tan^{-1}(3) \\
\tan^{-1}(8)+\cot^{-1}(-5) &=& \tan^{-1}(3)  \\
\tan^{-1}(13)+\cot^{-1}(-4) &=& \tan^{-1}(3)
\end{eqnarray}
So we have $6$ solutions.
But if we use the convention $\cot^{-1}y\in(0,\pi)$, we get
\begin{eqnarray}
\tan^{-1}(-7)+\cot^{-1}(-2) &=& \tan^{-1}(3) \\
\tan^{-1}(-2)+\cot^{-1}(-1) &=& \tan^{-1}(3) \\
\tan^{-1}(1)+\cot^{-1}(2) &=& \tan^{-1}(3)  \\
\tan^{-1}(2)+\cot^{-1}(7) &=& \tan^{-1}(3)  \\
\tan^{-1}(4)+\cot^{-1}(-13) &=& \tan^{-1}(3) +\pi\\
\tan^{-1}(5)+\cot^{-1}(-8) &=& \tan^{-1}(3) +\pi\\
\tan^{-1}(8)+\cot^{-1}(-5) &=& \tan^{-1}(3)  +\pi\\
\tan^{-1}(13)+\cot^{-1}(-4) &=& \tan^{-1}(3)+\pi
\end{eqnarray}
and we have only $4$ solutions.
</s>