<s>[INST] Calculate the density function of $Y=\frac{1}{X}-X$ where $X\sim U[0,1]$ I know that :
$$f_X(x)=\cases{1 & $x\in [0,1]$\\0 & $x\notin[0,1]$}$$
Then:
$$P(Y\leq y)=P(\frac{1}{X}-X\leq y)=P(X\leq\frac{1}{2}(\sqrt{y^2+4}-y))$$ as $$\frac{1}{x}-x=y\rightarrow x\frac{1}{x}-xx=yx\rightarrow 1-x^2=yx\rightarrow 1-x^2-xy=0\rightarrow \cases{\frac{1}{2}(\sqrt{y^2+4}-y)\\\frac{1}{2}(-\sqrt{y^2+4}-y)}  $$ I assumed positive root only(dont know if it is right assumption). So I am stuck from now on. Any suggestions.
 [/INST] As mentioned by @André and @Did, your inequality sign is the wrong way. We should have
$$F_Y(y)=
P(Y\leq y) = P\left(\dfrac{1}{X}-X \leq y \right) = P\left(X^2+yX-1 \geq 0 \right).
$$
You found the quadratic roots correctly. So the general solution to the quadratic inequality is $$X\leq \frac12\left(-y-\sqrt{y^2+4}\right)\quad\text{or}\quad X\geq\frac12 \left(-y+\sqrt{y^2+4}\right).$$ However, we must have $X\in (0,1)$ so we take the positive range only and we get, for any $y\gt 0$,
$$F_Y(y) = P\left(X\geq\frac12 \left(-y+\sqrt{y^2+4}\right)\right) = \dfrac{2+y-\sqrt{y^2+4}}{2}.$$
For $y\le0$, $F_Y(y)=0$. Differentiating $F_Y$ (and rearranging the result slightly) shows that the density $f_Y$ is $$f_Y(y) = \frac{\mathbf 1_{y>0}}{2}\left(1-\frac{y}{\sqrt{4+y^2}}\right) = \frac{\left(\sqrt{4+y^2}-y\right)\,\mathbf 1_{y>0}}{2\sqrt{4+y^2}} \cdot \frac{\sqrt{4+y^2}+y}{\sqrt{4+y^2}+y} = \frac{2\,\mathbf 1_{y>0}}{4+y^2+y\sqrt{4+y^2}}.$$
</s>