<s>[INST] What is the best way to solve an equation of the form $(f(x))^2-a(f(x))+b=x$? On a math contest I was told to solve the equation $$(x^2-3x+1)^2-3(x^2-3x+1)+1=x$$
For this particular problem I simplified by letting     $$a\equiv x^2-3x+1$$
Then I continued with $$a^2-3a+1-x=0$$
$$a=\frac{3\pm\sqrt{9-4\left(1-x\right)}}{2}$$
$$3\pm\sqrt{5+4x}=2a=2x^2-6x+2$$
$$\pm\sqrt{5+4x}=2x^2-6x-1$$
 I am not sure how to finish off this problem. Also, I have seen a bunch of problems like this in the past. What is the best way to approach a problem like this and also how could I finish solving this problem? Could someone also explain why f(f(x))=x has the same solutions as f(x)=x.
 [/INST] Let $f(x) = x^2-3x+1$.
Idea
The important observation here is following: If you have a solution $a$ that satisfies $f(a) = a$ then obviously
$a$ also solves the original equation because $f(f(a)) = f(a) = a$.
Finding the $a$ that solve $f(a) = a$ is easy, that is just a quadratic equation. This means you can divide $f(f(x))-x$ by $(x-a)$ for both solutions $a$ and you get a quadratic equation in $x$ which again can be solved easily.
Calculations
So lets first solve $f(x) = x$ This is aequivalent to $0 = x^2-4x+1$ which has the solutions $x_{1,2} = 2 \pm \sqrt{3}$
Now we can divide $f(f(x))-x$ by $(x-x_1)$ and $(x-x_2)$ or combined by $(x-x_1)(x-x_2) = x^2-4x+1 $.
Via polynomial division we get $$(f(f(x))-1)/(x^2-4x+1) = x^2-2x+1$$
Remember taht  $f(f(x))-x = 0$ is equivalent now to
$0=f(f(x))-x=(x-x_1)(x-x_2)(x^2-2x+1) = f(f(x))-x$.
As $x^2-2x+1$ has the zeros $x_{3,4} = 1 \pm \sqrt{2}$ we can further factorize
$0=f(f(x))-x=(x-x_1)(x-x_2)(x-x_3)(x-x_4)$ and we have all four solutions.
</s>