<s>[INST] Finding a function with certain properties I ran into a problem, and I'm not sure how to continue.
Problem: Let $f$ be a function such that $\sqrt {x - \sqrt { x + f(x) } } = f(x)$, for $x > 1$. In that domain, $f(x)$ has the form $\dfrac{a+\sqrt{cx+d}}{b}$, where $a$, $b$, $c$, $d$ are integers and $a$, $b$ are relatively prime. Find $a+b+c+d$.
So, I tried to cancel out the radicals, and got $(f(x))^4-2x(f(x))^2-f(x)+x^2-x=0$. Setting $y=f(x)$, I tried to apply the quadratic formula to find $x$ in terms of $y$. I got 
$$
x=\frac{x+2xy^2 \pm \sqrt{(x+2xy^2)^2-4(y^4-y)}}{2}. 
$$
From here, I tried simplifying the radical, but got 
$$
x=\frac{x+2xy^2 \pm \sqrt{(x^2)(2y^2+1)^2-y(4y^3+1)}}{2}.
$$
I don't know if I factored it wrong, or if I'm missing something painfully obvious. Can I have a hint as to how to continue? It would be greatly appreciated.
 [/INST] Putting $y=f(x)$, your equation becomes:
$y^4 - 2xy^2 + x^2 - x - y = 0$
This is hard to solve for $y$, but we have the suggestion that $y$ has the form $y=\frac{a+\sqrt{cx+d}}{b}$. This can be substituted into our polynomial and then simplified:
$\left(\frac{a+\sqrt{cx+d}}{b}\right)^4 - 2x\left(\frac{a+\sqrt{cx+d}}{b}\right)^2 + x^2 - x - \frac{a+\sqrt{cx+d}}{b} = 0$
$\implies\left(a+\sqrt{cx+d}\right)^4 - 2b^2x\left(a+\sqrt{cx+d}\right)^2 +b^4x^2 - b^4x - b^3\left(a+\sqrt{cx+d}\right) = 0$
$\implies a^4+4a^3\sqrt{cx+d}+6a^2(cx+d)+4a(cx+d)^{3/2}+(cx+d)^2-2a^2b^2x-4ab^2x\sqrt{cx+d}-2b^2x(cx+d)+b^4x^2-b^4x-ab^3-b^3\sqrt{cx+d}=0$
$\implies a^4+6a^2(cx+d)+(cx+d)^2-2a^2b^2x-2b^2x(cx+d)+b^4x^2-b^4x-ab^3=b^3\sqrt{cx+d}+4ab^2x\sqrt{cx+d}-4a^3\sqrt{cx+d}-4a(cx+d)^{3/2}$
$\implies a^4+6a^2(cx+d)+(cx+d)^2-2a^2b^2x-2b^2x(cx+d)+b^4x^2-b^4x-ab^3 = (b^3+4ab^2x-4a^3-4a(cx+d))\sqrt{cx+d}$
$\implies (a^4+6a^2(cx+d)+(cx+d)^2-2a^2b^2x-2b^2x(cx+d)+b^4x^2-b^4x-ab^3)^2 = (b^3+4ab^2x-4a^3-4a(cx+d))^2(cx+d)$
If you multiply this all out, and move everything to the left, you'll have a polynomial of the form $g(x)=M_4x^4 + M_3x^3 + M_2x^2 + M_1x + M_0 = 0$, where each of the $M_i$'s is a polynomial in $a$, $b$, $c$ and $d$. Since $g(x)=0$ for all $x>1$, we can say that $M_4=M_3=M_2=M_1=M_0=0$. If there is a solution to this system of five polynomials in four variables, then that's your answer.
In general, systems of polynomials can be difficult, but certain software packages allow for Groebner basis calculations, which can work, depending on the computer's capabilities and the user's patience.
One hopes there's an easier way to solve this, but the above is a sort of brute-force approach. 
EDIT The solution posted here by Hagen von Eitzen using limits is considerably nicer.
</s>