<s>[INST] $a,b$ are positive integers . Prove that if $k=\frac{a^2+b^2}{a+b+1}$ is an integer then $k=5$. This is the original question that was posted and answered. For the question the OP intended to ask, please see the edit history.

$a,b$ are positive integers. Prove that if $k=\frac{a^2+b^2}{a+b+1}$ is an integer then $k=5$.

I tried to see the whole expression as a quadratic of $a$ but that is not helping much.
 [/INST] In this case, if $k=5,$ then we would have $70$ as the sum of two integer squares, which it is not.
$$ a^2 + b^2 = 5a + 5b + 5,   $$
$$ 4a^2 + 4 b^2 = 20a + 20 b + 20, $$
$$ 4 a^2 - 20a + 4 b^2 - 20 b = 20, $$
$$ 4 a^2 - 20a + 25 + 4 b^2 - 20 b + 25 = 20 + 25 + 25 = 70, $$
$$ (2a-5)^2 + (2b-5)^2 = 70. $$
However, $70 = 2 \cdot 5 \cdot 7$ is not the sum of two integer squares. 
A superficially similar problem that does lead to $5$ is
$$ \frac{a^2 + b^2}{ab-1} = k, $$
see Is it true that $f(x,y)=\frac{x^2+y^2}{xy-t}$ has only finitely many distinct positive integer values with $x$, $y$ positive integers?  and my three or so answers there. I did, with help, eventually settle the whole thing.
If the version I have guessed is really the original problem, then it fits under https://en.wikipedia.org/wiki/Vieta_jumping
</s>