<s>[INST] Locating possible complex solutions to an equation involving a square root Note after reviewing answers. This question illustrates in a non-trivial way how the choice of how to compute the square root in the complex plane can make real difference to the answer you  get.
In this question finding all the solutions came down to solving:
$$ab-1=\sqrt{a^2+b^2+1}$$ subject to $$(a+1)(b+1)=1$$
The condition implies $a^2+b^2+1=(ab-1)^2$, but this leaves open the possibility of $\sqrt {a^2+b^2+1}=1-ab$ rather than $ab-1$.
The solution and discussion on the original question was getting rather long, and dealt mainly with real roots, so I've extracted this sub-question on complex roots.
Could someone please explain what constraints can be put on the complex solutions of this equation - the values of $a$ and $b$ - which would be consistent with the square root operation. It would help if answers were elementary and didn't assume a prior knowledge of complex square roots.
 [/INST] (A solution using too much algebra.)
I show that
the solutions are
$a=0, b=0$
and
$a=1, b=-1/2$
and,
in both of these,
the minus sign has to be taken
in the $\sqrt{}$.
If
$(a+1)(b+1)=1$,
then
$b = \dfrac1{a+1}-1$
so
$\begin{align}
ab-1
&=a(\frac1{a+1}-1)-1\\
&=a\frac1{a+1}-(a+1)\\
&=\frac{a-(a+1)^2}{a+1}\\
&=\frac{-a^2-a-1}{a+1}\\
&=-\frac{a^2+a+1}{a+1}\\
\end{align}
$
and
$\begin{align}
a^2+b^2+1
&=a^2+(\frac1{a+1}-1)^2+1\\
&=a^2+(\frac1{a+1})^2-2\frac1{a+1}+2\\
&=\frac{a^2(a+1)^2+1-2(a+1)+2(a+1)^2}{(a+1)^2}\\
&=\frac{(a^2+2)(a+1)^2+1-2(a+1)}{(a+1)^2}\\
&=\frac{(a^2+2)(a^2+2a+1)-2a-1}{(a+1)^2}\\
&=\frac{a^4+2a^3+a+2a^2+4a+2-2a-1}{(a+1)^2}\\
&=\frac{a^4+2a^3+2a^2+3a+1}{(a+1)^2}\\
\end{align}
$
Putting these in 
$ab-1=\sqrt{a^2+b^2+1}
$,
$-\dfrac{a^2+a+1}{a+1}
=\sqrt{\dfrac{a^4+2a^3+2a^2+3a+1}{(a+1)^2}}
$.
Multiplying both sides by
$a+1$,
$-(a^2+a+1)
=\sqrt{a^4+2a^3+2a^2+3a+1}
$.
Since
$(a^2+a+1)^2
=a^4+2a^3+3a^2+2a+1
$,
squaring and subtracting gives
$3a^2+2a = 2a^2+3a$
or
$a^2 = a$,
so
$a=0$
or $a=1$.
If $a=0$,
then $b=0$
and we have to take
the negative sign in the $\sqrt{}$.
If $a=1$,
then $b=-1/2$,
$ab-1 = -3/2$,
and
$a^2+b^2+1
=1+1/4+1
=9/4
= (3/2)^2
$,
and,
again,
we have to take
the negative sign in the $\sqrt{}$.
</s>