<s>[INST] Finding positive integer solutions to $n = ax^2 +by^2 - cxy$ How can I find the positive integer solutions to $x$ and $y$, given that $n$, $a$, $b$ and $c$ are all positive integers, in an equation of the form:
$$n = ax^2 + by^2 - cxy.$$
Specifically, I want to find the positive integer solutions to the following equation, given $n$:
$$n=3 x^2+20 y^2-16 x y.$$
 [/INST] $ax^2+by^2-cxy=n$
Expressing as a quadratic equation of x, $ax^2-x(cy)+by^2-n=0$
As x is positive  integer, $=>(cy)^2-4.a(by^2-n)=(c^2-4ab)y^2 +4an$ must be perfect square.
$c=16, a=3, b=20 =>16y^2+12n=d^2(say)$=>d is even=2e(say)
$=>4y^2+3n=e^2$=>e is odd $>e^2≡1(mod\ 8)$
$=>e^2-4y^2 \equiv 1 \pmod4$ => $3n \equiv 1 \pmod 4$=>$n\equiv -1 \pmod 4$ to admit solution.
So, n=4m-1 for some integer m.
Applying the the approach on y, $(c^2-4ab)x^2 +4bn$ must be perfect square,
or, $16x^2+80n=f^2$=>f must even=2g(say),
$=>4x^2+20n=g^2$=>g is even=2h(say),
$=>x^2+5n=h^2$
If h=5s for integer s, $x^2≡0(mod\ 5)$=>x=5t for some integer t.
If h=5s±1 for integer s, $x^2≡1(mod\ 5)$=>x=5t±1 for some integer t.
If h=5s±2 for integer s, $x^2≡4(mod\ 5)$=>x=5t±2 for some integer t.
n will be $\frac{h^2-x^2}{5}$
$=>x^2+5(4m-1)=h^2$
If x,h are both odd or both even, $5(4m-1)=h^2-x^2≡0(mod\ 4)$  which is impossible.
If x is even, $h^2=x^2+5(4m-1)≡-1(mod\ 4)$ which is impossible.
=> x must be odd and h must be even to admit solution.
y can also calculated from the given once x is known.
</s>