<s>[INST] How to obtain $f(x)$, if it is known that $f(f(x))=x^2+x$? How to get $f(x)$, if we know that $f(f(x))=x^2+x$?
Is there an elementary function $f(x)$ that satisfies the equation?
 [/INST] Here’s a technique for finding the first few terms of a formal power series representing the fractional iterate of a given function like $f(x)=x+x^2$. I repeat that this is a formal solution to the problem, and leaves unaddressed all considerations of convergence of the series answer.
I’m going to find the first six terms of $f^{\circ1/2}(x)$, the “half-th” iterate of $f$, out to the $x^5$-term. Let’s write down the iterates of $f$, starting with the zero-th.
\begin{align}
f^{\circ0}(x)&=x\\
f^{\circ1}=f&=x&+x^2\\
f^{\circ2}&=x&+2x^2&+2x^3&+x^4\\
f^{\circ3}&\equiv x&+3x^3&+6x^3& + 9x^4& + 10x^5& + 8x^6\\
f^{\circ4}&\equiv x &+ 4x^2& + 12x^3& + 30x^4& + 64x^5& + 118x^6\\
f^{\circ5}&\equiv x& + 5x^2& + 20x^3& + 70x^4& + 220x^5& + 630x^6\\
f^{\circ6}&\equiv x& + 6x^2& + 30x^3& + 135x^4& + 560x^5& + 2170x^6\\
f^{\circ7}&\equiv x& + 7x^2& + 42x^3& + 231x^4& + 1190x^5& + 5810x^6\,,
\end{align}
where the congruences are modulo all terms of degree $7$ and more.
Now look at the coefficients of the $x$-term: always $1$. Of the $x^2$-term? In $f^{\circ n}$, it’s $C_2(n)=n$. The coefficient of $x^3$ in $f^{\circ n}$ is $C_3(n)=n(n-1)=n^2-n$, as one can see by inspection. Now, a moment’s thought (well, maybe several moments’) tells you that $C_j(n)$, the coefficient of $x^j$ in $f^{\circ n}$, is a polynomial in $n$ of degree $j-1$. And a familiar technique of finite differences shows you that
\begin{align}
C_4(n)&=\frac{2n^3-5n^2+3n}2\\
C_5(n)&=\frac{3n^4-13n^3+18n^2-8n}3\,,
\end{align}
I won’t go into the details of that technique. The upshot is that, modulo terms of degree $6$ and higher, you have $f^{\circ n}(x)\equiv x+nx^2+(n^2-n)x^3+\frac12(2n^3-5n^2+3n)x^4+\frac13(3n^4-13n^3+18n^2-8n)x^5$.
Now, you just plug in $n=\frac12$ in this formula to get your desired series. And I’ll leave it to you to go one degree higher, using the iterates I’ve given you.
</s>