<s>[INST] Find maximum and minimum value of inverse function . 
We have to find maximum and minimum value of $x^2 +y^2$
My try 

how can I proceed
 [/INST] Let $\lfloor xy \rfloor=n$, then
$$\frac{x^2+y^2}{1-x^2y^2} = \tan \frac{n\pi}{4}$$

You've checked that no solutions for $0\le xy \le 1$, hence the tangent value must be negative which is achievable when $n=3,7,\ldots, 4k-1, \ldots$

Now
\begin{align}
  \frac{x^2+y^2}{1-x^2y^2} &= \tan \left( k\pi-\frac{\pi}{4} \right) \\
  x^2+y^2 &= x^2y^2-1
\end{align}
The required minimum value occurs when $xy=3$, therefore
$$\fbox{$\min \{ x^2+y^2 \}=8$}$$
and there're no maxima.
</s>