<s>[INST] Show that $\sqrt{a-b\sqrt{a+b\sqrt{a-b\sqrt{a+...}}}}=\sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}$ 
Assuming that $a>b^2$ show that
  $$\sqrt{a-b\sqrt{a+b\sqrt{a-b\sqrt{a+...}}}}=\sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}$$  (corrected)

This problem listed in a contest-math preparation book with the tag Russian-IMO-Longlist 1999.
It is not in the problem statement, but I'm assuming that $a,b \in \Bbb R$ and $a,b>0$. I tried two routes, without much progress: 
(a) developed $x=\sqrt{a-b\sqrt{a+bx}}$, as the solution $x$ must be a fixed point of $f(x)=\sqrt{a-b\sqrt{a+bx}}$, but the development does not appear promising (or the required algebra is beyond my capabilities).
(b) evaluated $f(x)=\sqrt{a-b\sqrt{a+bx}}$ on $x=\sqrt{a-\frac{3b^2}{2}}-\frac{b}{4}$ with the hope of getting $f(x)=\sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}$. Perhaps a naive approach. 
Not great results from both approaches.
Hints and solutions are welcomed. Sorry if this is a duplicate. 
 [/INST] Simplifying $x = \sqrt{a-b\sqrt{a+bx}}$ gives
\begin{equation*}
\frac{a^2}{b^3}-\frac{2 a x^2}{b^3}-\frac{a}{b}+\frac{x^4}{b^3}-x
= \frac{1}{b^3}(x^2+bx+b^2-a)(x^2-bx-a) = 0,
\end{equation*}
so that
\begin{equation*}
x = -\frac{b}{2}\pm\sqrt{a-\frac{3b^2}{4}}\text{ or }
x = \frac{b}{2}\pm\sqrt{a + \frac{b^2}{4}}.
\end{equation*}
The two possibilities with minus signs are negative (since $a>b^2$), which we reject since clearly $x>0$. Further, since $x < \sqrt{a}$, we also reject $x = \frac{b}{2} + \sqrt{a+\frac{b^2}{4}}$, leaving only
$$x = \sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}.$$
</s>