<s>[INST] Extremal problem with positive integer numbers Let $a,b$ be two positive integer numbers such that $a\sqrt{3}-b\sqrt{7}>0$. Find the minimum value of
$$
S=(a\sqrt{3}-b\sqrt{7})(a+b).
$$
Attempt I have tried and guess that the minimum value of $S$ is $(55+36)(55\sqrt{3}-36\sqrt{7})$, where $(55,36)$ is the integer solution of the Pell equation $3a^2-7b^2=3$.
 [/INST] Here is a sketch proof that the greatest lower bound for $S$ is $(3/2)(1/\sqrt3 + 1/\sqrt7) = 1.43297\ldots$. Call this $m$. Note that $(3 - \sqrt3m)/\sqrt3 = (\sqrt7m - 3)\sqrt7 = k$, say.
For integers $a,b$ such that $\sqrt3a - \sqrt7b > 0$ define
$$\delta(a,b) = (\sqrt3a - \sqrt7b)(a + b) - m.$$
As noted in the link, we can't have $3a^2 - 7b^2 = 1$ or $2$ (consider residues mod $3$ and $7$). But, as noted by Piyush Davyanakar, there are infinitely many $(a,b)$ such that $3a^2 - 7b^2 = 3$. We can get these more simply by $(a,b) = (1,0), (55, 36), (6049, 3960)$ and in general $a_{n+1} = 110a_n - a_{n-1}$, $b_{n+1} = 110b_n - b_{n-1}$.
This can be proved together with $3a_na_{n-1} - 7b_nb_{n-1} = 165$ by simultaneous induction.
Since $3a^2 - 7b^2 \ge 3$ we have
$$\delta(a,b)(\sqrt3a + \sqrt7b) \ge 3(a + b) - m(\sqrt3a + \sqrt7b) = (3 - \sqrt3m)a - (\sqrt7m - 3)b = k(\sqrt3a - \sqrt7b) > 0.$$
On the other hand,
$$\delta(a,b)(\sqrt3a + \sqrt7b)^2 = k(3a^2 - 7b^2)$$
and this equals $3k$ for infinitely many $(a,b)$. So $\delta(a,b)$ can be made as small as we like.
</s>