<s>[INST] Determine fractional transformation which maps $\mathbb{R}$ to $\mathbb{R}$ and $S^1$ to $S^1$ I want to determine all fractional transformation $F:\mathbb{C}\to\mathbb{C}$ which maps $\mathbb{R}$ on $\mathbb{R}$ and the unit circle on the unit circle. Let $\begin{pmatrix}a&b\\c&d
\end{pmatrix}$ be such a linear transformation. 
For $x\in\mathbb{R}$, we must have that $\frac{ax+b}{cx+d}\in\mathbb{R}.$ Then $$\frac{ax+b}{cx+d}=
\frac{\overline{ax+b}}{\overline{cx+d}}=\frac{\overline{a}x+\overline{b}}{\overline{c}x+\overline{d}}.
$$
Can we conclue from this that $\begin{pmatrix}a&b\\c&d\end{pmatrix}=\lambda\begin{pmatrix}\overline{a}&\overline{b}\\\overline{c}&\overline{d}\end{pmatrix}$ for some $\lambda\in\mathbb{C}$ met $|\lambda|=1$. Then also for $y\in S^1$, we have that $$\bigg|\frac{ay+b}{cy+d}\bigg|=1$$
how can I continue?
 [/INST] As you have noted in your comment, $F$ either fixes $1$ and $-1$ or swaps them.
Replacing $F$ by $-F$ if necessary, we can assume $F$ fixes $1$ and $-1$. Then we have
$$
\frac{a\cdot 1 + b}{c \cdot 1 + d} = 1 \quad\quad \frac{a\cdot -1 + b}{c \cdot -1 + d} = -1
$$
So:
$
a + b = c + d
$
and
$
b - a = c - d
$
Adding these equations gives $2b = 2c$ and subtracting them gives $2a = 2d$.
If $b = 0$, $F(z) = z$, otherwise we can divide through by $b$ and so assume $b = 1$. Thus the matrix of $F$ is either
$$
\begin{pmatrix}1&0\\
0 & 1\end{pmatrix} \quad\mbox{or} \quad
\begin{pmatrix}a&1\\
1 & a\end{pmatrix} 
$$
for some $a$. However, $F$ maps the unit circle to itself, and as $F$ maps circles to circles (or lines), this will hold iff $|F(i)| = 1$, i.e., iff
$
|ai + 1| = |i + a|
$
which holds iff
$
(ai+1)(\overline{ai+1}) = (i + a)(\overline{i + a})
$
but
$$
\begin{align*}
(ai+1)(\overline{ai+1}) &= a\overline{a} + (a - \overline{a})i + 1
\\
(i + a)(\overline{i + a}) &= a\overline{a} + (\overline{a} - a)i + 1
\end{align*}
$$
So $|F(i)| = 1$ iff $a - \overline{a} = 0$, i.e., iff $a \in \Bbb{R}$.
We conclude that the linear transformation $F$ maps $\Bbb{R}$ to $\Bbb{R}$ and $S^1$ to $S^1$ iff its matrix is either
$$
\pm \begin{pmatrix}1&0\\
0 & 1\end{pmatrix} \quad\mbox{or} \quad
\pm \begin{pmatrix}a&1\\
1 & a\end{pmatrix} 
$$
for some $a \in \Bbb{R}$. Note that this is degenerate when $a = 1$. For $a \neq 1$ it maps $\Bbb{R}$ onto $\Bbb{R}$ and $S^1$ onto $S^1$
</s>