<s>[INST] Determine all pairs of positive integers $(a,b)$ such that $a^2+b^2+ab$ is a perfect square. Consideration via mod $4$ shows that $a,b=0$ mod $4$ or one of between $a,b$ is $=1$ mod $4$ while the other is $=0 $ mod $4$.
Considering $(a+b)^2,a^2+an+b^2,(a+b-1)^2$ as we can deduce that $a,b>2$.
 [/INST] Just another way to approach the problem.
Since $$a^2+b^2+ab=(a+b)^2-ab $$ is a squared integer, there exists an integer $n$ such that
$$ (a+b)^2-(a+b-n)^2=ab$$
Solving it for $a$, we have
$$a = \frac{n (2 b - n)}{b - 2 n} $$
with the immediate condition  $b \neq 2 n$. However, we also have to set two other conditions. Because $a$ is positive, we must have $$ (2b-n)/(b-2n)>0$$ which for positive  $n$ leads to
$$b<\frac{n}{2} \,\,\,\text{or} \,\,\, b>2n$$
In addition, because $a$ is integer, we must have
$$(b-2n) |n (2 b - n)$$
Since $n(2b-n)$ can be written as $$n(2b-4n+3n)=2n(b-2n) +3n^2$$ it is divisible by $(b-2n)$ if and only if $(b-2n)$ divides $3n^2$. This occurs only when $b$ is of the form $$b=2n+k$$
where $k$ is a positive integer that divides $3n^2$.
Collecting all results, to find the pairs $a,b$ asked in the OP, we can:


*

*set an arbitrary positive integer $n$;

*take all positive integers $k$ that are divisors of $3n^2$;

*assign to $b$ the values $2n+k$;

*calculate for each $b$ the value $a=n(2b-n)/(b-2n)$.


For example, setting $n=1$, we get that the possible values of $k$, i.e. the divisors of $3 \cdot 1^2=3$, are $1$ and $3$. So we can assign to $b$ the values $2\cdot 1 +1=3$ and $2\cdot 1 +3=5$. The corresponding values of $a$ are $(6-1)/(3-2)=5$ and $(10-1)/(5-2)=3$. So the case $n=1$ yields the symmetric pairs $5,3$ and $3,5$. 
As another example, setting $n=6$, we get that the possible values of $k$, i.e. the divisors of $3 \cdot 6^2=108 $, are $1$, $2$, $3$, $4$, $6$, $9$, $12$, $18$, $27$, $36$, $54$ and $108$. Proceeding as above, we can assign to $b$ the values $2\cdot 6 +k$ and calculate the corresponding values of $a$. This yields the $a,b$ pairs $(120,13)$, $(66,14)$, $(48,15)$, $(39,16)$, $(30,18)$, $(24,21)$, plus the other six symmetric pairs. 
You can find here a Wolfram confirmation of the $12$ pairs generated by the case $n=6$.
</s>