<s>[INST] For what $n$ and $m$ is this number a perfect square? Find all positive integers $n$ and $m$ such that $$(2mn+1)^2-4mn(m+n)+n^2+m^2+(n-1)^2+(m-1)^2-3$$ is a perfect square.
Of course $n=m=1$ is a trivial solution and if you put $n=m+1$ (assume WLOG that $n \geq m$), the above expression is equal to $4n^4$ which is a perfect square - But I don't know if this is the only solution. I would really appreciate any help.
 [/INST] (Too long for a comment) I think the question can be made clearer with a rearrangement. The $2(2mn)(m+n)$ smells like a quadratic:
$$((m+n)-(2mn+1))^2=(2mn+1)^2-2(2mn+1)(m+n)+m^2+n^2+2mn$$
$$=(2mn+1)^2-4mn(m+n)-2(m+n)+m^2+n^2+2mn$$
Rearranging,
$$=(2mn+1)^2-4mn(m+n)+m^2+n^2+2mn-2(m+n)$$
$$=[(2mn+1)^2-4mn(m+n)+m^2+n^2]+2mn-2(m+n)$$
And substituting,
$$(2mn+1)^2-4mn(m+n)+m^2+n^2+(n-1)^2+(m-1)^2-3$$
$$=((m+n)-(2mn+1))^2-2mn+2(m+n)+(n-1)^2+(m-1)^2-3$$
$$=((m+n)-(2mn+1))^2-2mn+2m+2n+n^2-2n+m^2-2m-1$$
$$=((m+n)-(2mn+1))^2-2mn+n^2+m^2-1$$
$$=(m+n-2mn-1)^2+(n-m)^2-1$$
It's clear now why $n=m+1$ worked. It's equally clear that $m+n-2mn-1=1 \Rightarrow n=\frac{m-2}{2m-1}$ is a solution (although no positive integer solutions exist).
</s>