<s>[INST] How to solve the functional equation $ f(f(x))=ax^2+bx+c $ Find all real numbers $a,b,c\in\mathbb{R}$ for which there exists a function $f:\mathbb{R}\to\mathbb{R}$ such that:
$$
f(f(x))=ax^2+bx+c
$$
for all $x\in\mathbb{R}$.
The only thing I could deduce is:
$$
f(ax^2+bx+c)=af(x)^2+bf(x)+c
$$
Which doesn't help much. How to tackle the problem?
 [/INST] Here are some reflections for continuous and differentiable $f$. It turns out that there is a unique solution if some conditions are satisfied (you can look at the end for the final answer).



*

*$$
f(f(x))=ax^2+bx+c\implies f(ax^2+bx+c)=af(x)^2+bf(x)+c
$$

*If $f(x)=f(y)$, $x\neq y$ then:
$$
a(x+y)=b.
$$
If $a(x+y)=b$ then in any case $ff(x)=ff(y)$ which means that either $f(x)=f(y)$ or $a(f(x)+f(y))=b$. In any case there are two non-equal numbers $x'$ and $y'$ such that $f(x')=f(y')$ and they lie on two different sides of $\frac{-b}{2a}$. 

*If $f(x)=x$ then: $f(x)=ax^2+bx+c$ hence:
$$
ax^2+(b-1)x+c=0\implies (b-1)^2\geq 4ac; x=\frac{-b+1\pm\sqrt{(b-1)^2-4ac} }{2a}.
$$

*$f'(ax^2+bx+c)(2ax+b)=f'(x)(2af(x)+b)$ hence:
$$
x=\frac{-b}{2a}\implies f'(\frac{-b}{2a})=0\text{ or } f(\frac{-b}{2a})=\frac{-b}{2a}$$
On the other hand:
$$
f'(f(x)).f'(x)=2ax+b.
$$
So:


If $f'(x)=0$ then $x=\frac{-b}{2a}$. 

Note that from large enough $x$ and $y$ satisfying $a(x+y)=b$ and $f(x)=f(y)$


*

*If $f(\frac{-b}{2a})=\frac{-b}{2a}$, then $f\circ f(\frac{-b}{2a})=\frac{-b}{2a}$ which means:
$$
c-\frac{b^2}{4a}=-\frac b{2a}\implies b^2-4ac=2b.
$$


moreover:
$$
f'(f(\frac{-b}{2a})).f'(\frac{-b}{2a})=\implies f'(\frac{-b}{2a})=0.
$$
If $b^2-4ac\neq 2b$, $f'(\frac{-b}{2a})=0$; so in any case:

$$
f'(\frac{-b}{2a})=0.
$$

This means that the function is strictly increasing on the one side of $\frac{-b}{2a}$ and strictly decreasing on the other side.
Moreover if $f'(f(x))=0$ then $x=\frac{-b}{2a}$; so if there is $x$ such that $f(x)=\frac{-b}{2a}$ then $f'(f(x))=0$ which means that $x=\frac{-b}{2a}$. So:

$$f(\frac{-b}{2a})=\frac{-b}{2a}\implies b^2-4ac=2b$$

Without this condition there will be no answer.


*

*But if $a(x+y)=b$ and $f(x)\neq f(y)$, then $a(f(x)+f(y))=b$; but this means that both of them cannot be in the same time bigger (or smaller) than $\frac{-b}{2a}$; which is a contradiction since $f'(\frac{-b}{2a})=0$ and $\frac{-b}{2a}$ is an extremum point of $f$. Hence:


$$ f(x)=f(y), x\neq y\iff a(x+y)=b$$

Therefore it is enough to find the function for $x>\frac{-b}{2a}$. 


*

*See that after using the information derived above:
$$
f(ax^2+bx+\frac{b^2-2b}{4a})=af(x)^2+bf(x)+\frac{b^2-2b}{4a}\implies\\
f\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)=a\left(f(x)+\frac{b}{2a}\right)^2-\frac{b}{2a}.
$$.
Note that if $a>0$, then $x>-\frac{b}{2a}\implies f(x)>-\frac{b}{2a}$; w.l.o.g. we assume $a>0$ and $x>-\frac b{2a}$. 
$$
af\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)+\frac{b}{2}=\left(af(x)+\frac{b}{2}\right)^2
$$
See that defining $g(x)=af(x)+\frac b2$, we get:
$$
g(a(x+\frac{b}{2a})^2-\frac{b}{2a})=g(x)^2
$$
Define the function:
$$
h(x)=\frac{\log g(x)}{\log(ax+\frac b2)}.
$$
Then we get:
$$
h(a(x+\frac{b}{2a})^2-\frac{b}{2a})=h(x)
$$
This function can be shown to be a constant by constructing  a decreasing sequence $u_n$  from an arbitrary $u$ and showing that $h(u)$ is equal to the limit of $u_n$ which is a constant value independent of $u$. Hence:
$$
\frac{\log g(x)}{\log(ax+\frac b2)}=C\implies g(x)=(ax+\frac b2)^C\\
\implies f(x)=\frac 1a\left((ax+\frac b2)^C-\frac b2\right).
$$ 
By plugging in to the original question, it turns out that $C=\sqrt 2$. Extending it for all $x$'s, we get:



$$
f(x)=\frac 1a\left(|ax+\frac b2|^{\sqrt 2}-\frac b2\right) \text{ if } b^2-4ac=2b.
$$

</s>