<s>[INST] Find all integer solutions of $(a+b+3)^2+2ab=3ab(a+2)(b+2)$ Here is another number theory problem that I am not able to do
Find all integer solutions of $$(a+b+3)^2+2ab=3ab(a+2)(b+2)$$
My attempt$:$
On expanding we get,
$$b^2+4ab+6b+a^2+6a+9=3a^2b^2+6a^2b+6ab^2+12ab$$ or $$-6ab^2+b^2-8ab+6b+a^2+6a+9=3a^2b^2+6a^2b$$ or $$-6ab^2+b^2-8ab+6b+a^2+6a+9-3a^2b^2-6a^2b=0$$ or $$\left(1-3b^2-6b\right)a^2+\left(-6b^2-8b+6\right)a+b^2+6b+9=0$$ or $$a=\frac{-\left(-6b^2-8b+6\right)\pm \sqrt{\left(-6b^2-8b+6\right)^2-4\left(1-3b^2-6b\right)\left(b^2+6b+9\right)}}{2\left(1-3b^2-6b\right)}$$ or $$a=\frac{-\left(-6b^2-8b+6\right)\pm \:4\sqrt{3}\left(b+1\right)\sqrt{b\left(b+2\right)}}{2\left(1-3b^2-6b\right)};\quad \:b\ne \:-\frac{3+2\sqrt{3}}{3},\:b\ne \frac{2\sqrt{3}-3}{3}$$ and after further solving $$a=\frac{3b^2+4b-3+2\sqrt{3}b\sqrt{b\left(b+2\right)}+2\sqrt{3}\sqrt{b\left(b+2\right)}}{1-3b^2-6b}$$ and $$a=\frac{3b^2+4b-3-2\sqrt{3}b\sqrt{b\left(b+2\right)}-2\sqrt{3}\sqrt{b\left(b+2\right)}}{1-3b^2-6b}$$ where $b\ne \:-\frac{3+2\sqrt{3}}{3},\:b\ne \frac{2\sqrt{3}-3}{3}$
So, now I can assign some integer values to $a$ and check if that value of $a$ is giving some integer values of $b$ or not. Like if $a=0$ then $b=-3$. But I think it is merely hit and trial because this thing can also apply to the original equation. Any help will be greatly appreciated.
 [/INST] If $S=a+b$, $P=ab$ we have
$$S^2-6(P-1)S-(3P^2+10P-9)=0\rightarrow$$
$$S_{1,2}=3(P-1)\pm2\sqrt{\Delta}$$
$$\Delta=P(3P-2)$$
Now, we must find $P_i$ such as $\Delta$ be a square: $\Delta=K^2$. That is a difficult. I found only
$P_1=0\rightarrow\Delta=0\rightarrow S=-3$
$P_2=1\rightarrow\Delta=1\rightarrow S=\pm2$
$P_2=9\rightarrow\Delta=225\rightarrow S=24\pm15$
Now, for each pear $(S, P)$ solve $x^2-Sx+P=0$.
I don't know a theorem from the number theory to find all roots of the eq.
$$P(3P-2)=K^2$$
One attempt is
$$3P^2-2P=K^2\rightarrow 2P(P-1)=(K-P)(K+P)$$
but is not easier.
</s>