<s>[INST] Finding value of 1 variable in a 3-variable $2^{nd}$ degree equation The question is: If $a,b,\space (a^2+b^2)/(ab-1)=q$ are positive integers, then prove that $q=5$. Also prove that for $q=5$ there are infinitely many solutions in $\mathbf N$ for $a$ and $b$. I simplified the equation as follows:-$$\frac {a^2+b^2}{ab-1}=q\\\begin{align}\\&=>\frac {2a^2+2b^2}{ab-1}=2q\\&=>\frac{a^2+b^2+2ab+a^2+b^2-2ab}{ab-1}=2q\\&=>(a+b)^2+(a-b)^2=2q(ab-1)\\&=>2(a+b)^2+2(a-b)^2=q(4ab-4)\\&=>2(a+b)^2+2(a-b)^2=q((a+b)^2-(a-b)^2-4)\end{align}$$Substituting $a+b=X$ and $a-b=Y$, we get $$2X^2+2Y^2=q(X^2-Y^2-4)\\\begin{align}&=>(q-2)X^2=(q+2)Y^2+4q\end{align}$$Now using the quadratic residues modulo $5$, I know that $X^2,Y^2\equiv0, \pm1(mod\space 5)$. But using this directly doesn't give the answer. So what to do after this? An answer without the use of co-ordinate geometry would be greatly appreciated as it seems there is a very good resemblance of the equation to a pair of hyperbolas which are symmetric with respect to the line $y=x$ but I don't understand co-ordinate geometry very well. 
 [/INST] I take it that you have looked into the details of Vieta Jumping wiki link, so if we know the curve $x^2+y^2 - qxy + q = 0$ has a single integer solution then there are infinitely many, generated by the pair of iterations:
$$(x,y) \mapsto (qx-y,x)$$ $$(x,y) \mapsto (y,x)$$
on the graph of the curve.
Further observe that for a lattice point $(x,y)$ with $y > x >0$, the orbit of the iteration $(x,y) \mapsto (qx-y,x)$ does not change the branch of hyperbola, i.e., remains in the branch in the first quadrant.
Geometrically this is equivalent to moving from a lattice point $(x_0,y_0)$ to the point $(x_0,x_0)$ on the line $y = x$ and then join and extend the line joining $(x_0,x_0)$ and $(y_0,x_0)$ to meet the curve again at $(qx_0 - y_0,x_0)$.A sequence of iterations that will look like: 

Now, $x^2+y^2 - qxy + q = 0$ has no solution in natural numbers if $q = 1,2$
since,$x^2+y^2 - qxy + q > 0$ for $q = 1,2$.Thus, $q \ge 3$.
We note that the vertex of the branch in concern $\color{blue}{A} = \left(\sqrt{\frac{q}{q-2}},\sqrt{\frac{q}{q-2}}\right)$ lies on the line joining $\color{black}{J} = (1,1)$ and $\color{black}{H} = (2,2)$ for $q \ge 3$.
If $I = (x_i,y_i)$ is the lattice point in the orbit of the iteration closest to the vertex $\color{blue}{A}$, then the next point in the orbit $K$ no longer stays in the upper half (region $y>x$) part of the hyperbola, $K = (x_k,y_k)$ must lie in the lower half region ($x > y$).
But in order to go from $I$ to $K$ we find that the point on the line $x = y$, i.e., $(y_k,y_k)$ must be a lattice point as well. The only lattice point on this line between the origin and vertex $\color{blue}{A}$ is $J = (1,1)$, hence $y_k = 1$.
Therefore, both the roots of the quadratic $x^2+1 - qx + q = 0$ must be positive integers, viz $x_k$ and $qx_k - 1$.
The discriminant $\Delta = q^2 - 4(q+1)$ must be a perfect square. But $\Delta$ differs from $(q-2)^2$ by $8$, and the only perfect squares that differ by $8$ are $1$ & $9$.
$$\implies (q-2)^2 = 9 \implies  q = 5 \textrm{ since } q \ge 3$$
</s>