<s>[INST] Solving $a(a + 1) + b(b + 1) = 12, (a+ 1)(b+1) = 4$, is there any trick? What are all (possibly complex) solutions to the following two equations?
$a(a + 1) + b(b + 1) = 12$
$(a+ 1)(b+1) = 4$
Clearly one can substitute the second equation into the first then solve the quartic. But I am wondering if there is a shorter/sneakier way to solve this.
 [/INST] How sneaky would you like to be?  If we interpret these expressions as the curve equations $ \ x(x + 1) + y(y + 1) \ = \ 12 \ $ and $ \ ( x + 1)(y+1) \ = \ 4 \ \ , $ we can write the first as the circle equation $ \ \left(x + \frac12 \right)^2 + \left(y + \frac12 \right)^2 \ = \ \frac{25}{2} \ \ $ , with center at $ \ \left(-\frac12 \ , \ -\frac12 \right) \ $ and the second as a "translated" rectangular hyperbola with its center at $ \ (-1 \ , \ -1) \ \ . $  This may not seem helpful initially, but the location of the centers suggests that there is a "diagonal symmetry" for the solutions.
Two of these can be "picked off" by inspection from looking at the intercepts of the curves.  Setting $ \ x \ = \ 0 \ $ gives us $ \ 1·(y + 1) \ = \ 4 \ \Rightarrow \ y \ = \ 3 \ $ for the hyperbola, and we see that this also applies to the circle:  $ \ 0·(0 + 1) + 3·(3 + 1) \ = \ 12 \ \ . $  The symmetry indicates that there is also a solution  $ \ x \ = \ 3 \ , \ y \ = \ 0 \ \ . $  So the two curves intersect one another on the coordinate axes.
Are there any more solutions?  This symmetry about the line $ \ y = x \ $ suggests that we might look at $ \ (x + 1)^2 \ = 4 \ $ and $ \ 2·x(x + 1) \ = \ 12 \ \ . $  The first of these produces $ \ x \ = \ -1 \ \pm \ 2 \ = \ 1 \ , \ -3 \ \ ; \ $  the point $ \ (1 \ , \ 1) \ $ is where the diagonal line meets the hyperbola, but it is certainly not on the circle, while the other intersection is $  \ (-3 \ , \ -3) \ \ . $  The intersections of the diagonal line and circle are found from $$ \ x·(x + 1) \ = \ 6 \ \Rightarrow \ x^2 + x - 6 \ = \  (x + 3)·(x - 2) \ = \ 0 \ \ . $$  The point $ \ (2 \ , \ 2) \ $ is not on the hyperbola, but $  \ (-3 \ , \ -3) \  $ definitely is.  So the circle, hyperbola, and diagonal line all meet at $  \ (-3 \ , \ -3) \ \ .  $
The circle and hyperbola in fact just intersect tangentially there, which tells us that this solution has an even multiplicity.  In regard to the quartic equation you mention and which player3236 develops (a biquadratic), we can conclude that we have found all four of its solutions:  $ \ (0 \ , \ 3) \ \ , \ \ (3 \ , \ 0) \ \ ,    $ and $ \ (-3 \ , \ -3) \ $ with multiplicity $ \ 2 \ \ . $

We can check up on this by using the "rotated" coordinates $ \ u \ = \ x + y \ , \ v \ = \ x - y \ \ , $ which transform the curve equations into $ \ u^2 + 4u - v^2 \ = \ 12 \ , \ u^2 + 2u + v^2 \ = \ 24 \ \ . $  Adding these together yields
$$ 2u^2 \ + \ 6u \ - 36 \ \ = \ \ 2·(u + 6)·(u - 3) \ \ = \ \ 0 \ \ . $$
Solving each of the transformed curve equations for $ \ v^2 \ $ gives us
$$ v^2 \ \ = \ \ u^2  +  4u  -  12 \ \ = \ \ (u + 6)·(u - 2) \ \ \Rightarrow \ \ u \ = \ -6 \ \rightarrow \ v^2 \ = \ 0 \ \ , \ \  u \ = \ 3 \ \rightarrow \ v^2 \ = \ 9 $$
and
$$ v^2 \ \ = \ \ 24  -  u^2  -  2u  \ \ = \ \ -(u + 6)·(u - 4) \ \ \Rightarrow \ \ u \ = \ -6 \ \rightarrow \ v^2 \ = \ 0 \ \ , $$ $$  u \ = \ 3 \ \rightarrow \ v^2 \ = \ 9 \ \ . $$
So we find three solution points:
$$ (u \ , \ v) \ \rightarrow \ \left(x = \frac{u + v}{2} \ , \ y = \frac{u - v}{2} \right) \ : \ \ \ (3 \ , \ 3 ) \ \rightarrow \ (3 \ , \ 0 ) \ \ , \ \ (3 \ , \ -3 ) \ \rightarrow \ (0 \ , \ 3 ) \ \ , $$
$$ (-6 \ , \ 0 ) \ \rightarrow \ (-3 \ , \ -3 ) \ \ . $$
</s>